This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Build & Packaging

Prepare PostgreSQL rpm/deb pacakge building environment, and some packaging hints

1: Building Environment
2: Prepare rust & pgrx
3: Build PLv8
4: Build PostgresML
5: Build ParadeDB
6: Build DuckDB FDW
7: Build Parquet S3 FDW

1 - Building Environment

How to prepare VM env for building PostgreSQL RPM/DEB extensions, EL 8/9, Debian12, Ubuntu22

VM

To build PGML RPM packages in EL / Debian environment, you need to prepare a virtual machine environment. Pigsty provides a ext.yml template that can be used to prepare the virtual machine environment required for building.

cd pigsty
make build
./node.yml -i files/pigsty/build-ext.yml -t node_repo,node_pkg

It will launch four virtual machines with EL8, EL9, Debian12, and Ubuntu22 respectively, and install the necessary dependencies for building.

Proxy

If you are in a network environment that requires a proxy, you need to configure the proxy environment variables.

Here we assume that you have a proxy server available in your local environment: http://192.168.0.106:8118 (replace with your OWN proxy server).

PROXY=http://192.168.0.106:8118
export HTTP_PROXY=${PROXY}
export HTTPS_PROXY=${PROXY}
export ALL_PROXY=${PROXY}
export NO_PROXY="localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.*,*.myqcloud.com,*.tsinghua.edu.cn"
alias build="HTTPS_PROXY=${PROXY} cargo pgrx package -v"

Adding the following lines to ~/.ssh/config will make git clone via HTTPS proxy (optional):

Host github.com
    Hostname ssh.github.com
    Port 443
    User git

If the following command can connect to GitHub, then the proxy is correctly configured:

ssh -T [email protected]

Alias

You can use the following Alias to switch between different PG versions environment in the EL environment:

alias pg16="export PATH=/usr/pgsql-16/bin:/root/.cargo/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
alias pg15="export PATH=/usr/pgsql-15/bin:/root/.cargo/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
alias pg14="export PATH=/usr/pgsql-14/bin:/root/.cargo/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
alias pg13="export PATH=/usr/pgsql-13/bin:/root/.cargo/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"
alias pg12="export PATH=/usr/pgsql-12/bin:/root/.cargo/bin:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin"

And the following Alias for Debian environment:

alias pg16="export PATH=/usr/lib/postgresql/16/bin:/home/vagrant/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;"
alias pg15="export PATH=/usr/lib/postgresql/15/bin:/home/vagrant/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;"
alias pg14="export PATH=/usr/lib/postgresql/14/bin:/home/vagrant/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;"
alias pg13="export PATH=/usr/lib/postgresql/13/bin:/home/vagrant/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;"
alias pg12="export PATH=/usr/lib/postgresql/12/bin:/home/vagrant/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin;"

Tools

You’ll have to group install additional 'Development Tools' components in EL 8 / EL 9 environment. In EL8, you need to add the --nobest option to complete the installation due to dependency errors.

sudo yum groupinstall --nobest -y 'Development Tools';
rpmdev-setuptree  # setup rpmbuild environment

2 - Prepare rust & pgrx

It is necessary to install rust and pgrx before building Rust extensions.

Install Rust

To build Rust extensions, you need to install rust and pgrx

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"

Install pgrx, beware of version

cargo install --locked cargo-pgrx@${PGRX_VER-'0.11.3'}
cargo pgrx init

Rust Extension List

Vendor	Name	Version	PGRX	License	PG Ver	Deps
PostgresML	pgml	v2.9.3	v0.11.3	MIT	16,15,14
ParadeDB	pg_search	v0.8.6	v0.11.3	AGPLv3	16,15,14,13,12
ParadeDB	pg_lakehouse	v0.8.6	v0.11.3	AGPLv3	16,15
Supabase	pg_graphql	v1.5.7	v0.11.3	Apache-2.0	16,15
Supabase	pg_jsonschema	v0.3.1	v0.11.3	Apache-2.0	16,15,14,13,12
Supabase	wrappers	v0.4.1	v0.11.3	Apache-2.0	16,15,14
Tembo	pgmq	v1.2.1	v0.11.3	PostgreSQL	16,15,14,13,12
Tembo	pg_vectorize	v0.17.0	v0.11.3	PostgreSQL	16,15,14	pgmq, pg_cron
Tembo	pg_later	v0.1.1	v0.11.3	PostgreSQL	16,15,14,13	pgmq
VADOSWARE	pg_idkit	v0.2.3	v0.11.3	Apache-2.0	16,15,14,13,12
pgsmcrypto	pgsmcrypto	v0.1.0	v0.11.3	MIT	16,15,14,13,12
kelvich	pg_tiktoken	v0.0.1	v0.10.2	Apache-2.0	16,15,14,13,12
rustprooflabs	pgdd	v0.5.2	v0.10.2	MIT	16,15,14,13,12
timescale	vectorscale	v0.2.0	v0.11.4	PostgreSQL	16,15,14,13,12
kaspermarstal	plprql	v0.1.0	v0.11.4	Apache-2.0	16,15,14,13,12

Download

cd ~;
cd ~; git clone --recursive [email protected]:postgresml/postgresml.git  ; cd ~/postgresml     && git checkout v2.9.3
cd ~; git clone --recursive https://github.com/paradedb/paradedb.git  ; cd ~/paradedb       && git checkout v0.8.6
cd ~; git clone [email protected]:supabase/pg_graphql.git                ; cd ~/pg_graphql     && git checkout v1.5.7                 
cd ~; git clone [email protected]:supabase/pg_jsonschema.git             ; cd ~/pg_jsonschema  && git checkout v0.3.1                    
cd ~; git clone [email protected]:supabase/wrappers.git                  ; cd ~/wrappers       && git checkout v0.4.1               
cd ~; git clone [email protected]:tembo-io/pgmq.git                      ; cd ~/pgmq           && git checkout v1.2.1 #v1.3.3                        
cd ~; git clone [email protected]:tembo-io/pg_vectorize.git              ; cd ~/pg_vectorize   && git checkout v0.17.0 
cd ~; git clone [email protected]:tembo-io/pg_later.git                  ; cd ~/pg_later       && git checkout v0.1.1               
cd ~; git clone [email protected]:VADOSWARE/pg_idkit.git                 ; cd ~/pg_idkit       && git checkout v0.2.3               
cd ~; git clone [email protected]:zhuobie/pgsmcrypto.git 
cd ~; git clone [email protected]:kelvich/pg_tiktoken.git
cd ~; git clone [email protected]:rustprooflabs/pgdd.git                 ; cd ~/pgdd           && git checkout 0.5.2           
cd ~; git clone [email protected]:kaspermarstal/plprql.git               ; cd ~/plprql         && git checkout v0.1.0             
cd ~; git clone [email protected]:timescale/pgvectorscale.git            ; cd ~/pgvectorscale  && git checkout 0.2.0                    

cd ~/paradedb;     cargo update
cd ~/pgmq/pgmq-rs; cargo update

Build

cd ~/paradedb/pg_search;       pg16 build;    pg15 build;    pg14 build;    pg13 build;  pg12 build; 
cd ~/paradedb/pg_lakehouse;    pg16 build;    pg15 build;                    
cd ~/pg_graphql;               pg16 build;    pg15 build;    pg14 build;     
cd ~/pg_jsonschema;            pg16 build;    pg15 build;    pg14 build;    pg13 build;  pg12 build; 
cd ~/wrappers/wrappers;        pg16 build;    pg15 build;    pg14 build;     
cd ~/pgmq;                     pg16 build;    pg15 build;    pg14 build;    pg13 build;  pg12 build; 
cd ~/pg_tier;                  pg16 build;                                   
cd ~/pg_vectorize/extension;   pg16 build;    pg15 build;    pg14 build;                
cd ~/pg_later;                 pg16 build;    pg15 build;    pg14 build;    pg13 build; 
cd ~/pgsmcrypto;               pg16 build;    pg15 build;    pg14 build;    pg13 build;  pg12 build; 
cd ~/pg_idkit;                 pg16 build;    pg15 build;    pg14 build;    pg13 build;  pg12 build; 
cd ~/plprql/plprql;            pg16 build;    pg15 build;    pg14 build;    pg13 build;  pg12 build; 

export RUSTFLAGS="-C target-feature=+avx2,+fma" cd ~/pgvectorscale/pgvectorscale;
pg16 build;    pg15 build;

# pgrx 0.10.2
cd ~/pgdd;                     pg16 build;    pg15 build;    pg14 build;       # 16,15,14
cd ~/pg_tiktoken;              pg16 build;    pg15 build;    pg14 build;       # 16,15,14

3 - Build PLv8

Build the Javascript procedural language extension PLv8 for PostgreSQL.

Build plv8

GitHub Repo: PLv8

Install Deps (EL)

sudo yum groupinstall --nobest -y 'Development Tools'; 
sudo yum install git cmake

Clone the source

rpmdev-setuptree
cd ~/rpmbuild/SOURCES/
git clone [email protected]:plv8/plv8.git

cd plv8; git checkout v3.2.2
make -j16

Make RPM:

rpmbuild --define "pgmajorversion 16"  -ba ~/rpmbuild/SPECS/plv8.spec
rpmbuild --define "pgmajorversion 15"  -ba ~/rpmbuild/SPECS/plv8.spec
rpmbuild --define "pgmajorversion 14"  -ba ~/rpmbuild/SPECS/plv8.spec
rpmbuild --define "pgmajorversion 13"  -ba ~/rpmbuild/SPECS/plv8.spec
rpmbuild --define "pgmajorversion 12"  -ba ~/rpmbuild/SPECS/plv8.spec

Or:

cd ~/rpmbuild/SPECS && make plv8

4 - Build PostgresML

There are some extra settings to build PostgresML on EL8 and EL9.

It’s quite tricky to build PostgresML on EL8 and EL9, here is the tutorial.

First of all, configure the RPM build environment, install rust and pgrx according to the instructions.

Python3

Install Python and setup alternatives:

sudo yum install python3.11 python3.11-devel python3-virtualenv openssl openssl-devel cmake pkg-config libomp libomp-devel openblas* llvm llvm-devel lld openblas*
sudo alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1
sudo alternatives --set python3 /usr/bin/python3.11
sudo alternatives --set python /usr/bin/python3.11

Clone Repo

Clone pgml with git then checkout to the latest release:

cd ~; git clone --recursive [email protected]:postgresml/postgresml.git; 
cd ~/postgresml && git checkout v2.9.3
cd ~/postgresml/pgml-extension

EL8 Ad hoc

This part is only for EL8, EL9 is not affected.

sudo dnf install gcc-toolset-13
source /opt/rh/gcc-toolset-13/enable

source /opt/rh/gcc-toolset-13/enable
export CC=/opt/rh/gcc-toolset-13/root/usr/bin/gcc
export CXX=/opt/rh/gcc-toolset-13/root/usr/bin/g++
export LD_LIBRARY_PATH=/opt/rh/gcc-toolset-13/root/usr/lib64:$LD_LIBRARY_PATH

You have to change build.rs, add stdc++fs and gcc-toolset-13:

println!("cargo:rustc-link-lib=static=stdc++fs");
println!("cargo:rustc-link-search=native=/opt/rh/gcc-toolset-13/root/usr/lib/gcc/x86_64-redhat-linux/13");

The whole file should be like this:

fn main() {
    #[cfg(target_os = "macos")]
    {
        println!("cargo:rustc-link-search=/opt/homebrew/opt/openblas/lib");
        println!("cargo:rustc-link-search=/opt/homebrew/opt/libomp/lib");
    }

    // PostgreSQL is using dlopen(RTLD_GLOBAL). this will parse some
    // of symbols into the previous opened .so file, but the others will use a
    // relative offset in pgml.so, and will cause a null-pointer crash.
    //
    // hide all symbol to avoid symbol conflicts.
    //
    // append mode (link-args) only works with clang ld (lld)
    println!(
        "cargo:link-args=-Wl,--version-script={}/ld.map",
        std::env::current_dir().unwrap().to_string_lossy(),
    );

    println!("cargo:rustc-link-lib=static=stdc++fs");
    println!("cargo:rustc-link-search=native=/opt/rh/gcc-toolset-13/root/usr/lib/gcc/x86_64-redhat-linux/13");

    vergen::EmitBuilder::builder().all_git().emit().unwrap();
}

Then change the Cargo.toml

Add cc = "1.0" to the [build-dependencies] section:

[build-dependencies]
+++ cc = "1.0"

Building

To build PostgresML against PostgreSQL 16, 15, 14:

cd ~/postgresml/pgml-extension; pg16 build; pg15 build; pg14 build;

The rpm package will be placed in ~/rpmbuild/RPMS/x86_64/ directory.

rm -rf ~/rpmbuild/SOURCES/pgml_16; cp -r ~/postgresml/pgml-extension/target/release/pgml-pg16 ~/rpmbuild/SOURCES/pgml_16;
rm -rf ~/rpmbuild/SOURCES/pgml_15; cp -r ~/postgresml/pgml-extension/target/release/pgml-pg15 ~/rpmbuild/SOURCES/pgml_15;
rm -rf ~/rpmbuild/SOURCES/pgml_14; cp -r ~/postgresml/pgml-extension/target/release/pgml-pg14 ~/rpmbuild/SOURCES/pgml_14;

cd ~/rpmbuild/SPECS && make pgml

# or build theme manually:
rm -rf ~/rpmbuild/RPMS/x86_64/pgml*.rpm;
rpmbuild --without debuginfo --define "pgmajorversion 16" -ba ~/rpmbuild/SPECS/pgml.spec
rpmbuild --without debuginfo --define "pgmajorversion 15" -ba ~/rpmbuild/SPECS/pgml.spec
rpmbuild --without debuginfo --define "pgmajorversion 14" -ba ~/rpmbuild/SPECS/pgml.spec

5 - Build ParadeDB

There are some important ParadeDB extensions that need to be built and packaged.

Setup the environment, rust 与 pgrx

Beware of the pgrx version.

Clone paradedb repo and checkout the latest version:

cd ~; git clone --recursive [email protected]:paradedb/paradedb.git;
cd ~/paradedb; git checkout v0.8.6

pg_search

cd ~/paradedb/pg_search/
cargo update

# build pg_search
pg16 build
pg15 build
pg14 build
pg13 build
pg12 build

# move to rpmbuild SOURCES dir
rm -rf ~/rpmbuild/SOURCES/pg_search_16; cp -r ~/paradedb/target/release/pg_search-pg16 ~/rpmbuild/SOURCES/pg_search_16;
rm -rf ~/rpmbuild/SOURCES/pg_search_15; cp -r ~/paradedb/target/release/pg_search-pg15 ~/rpmbuild/SOURCES/pg_search_15;
rm -rf ~/rpmbuild/SOURCES/pg_search_14; cp -r ~/paradedb/target/release/pg_search-pg14 ~/rpmbuild/SOURCES/pg_search_14;
rm -rf ~/rpmbuild/SOURCES/pg_search_13; cp -r ~/paradedb/target/release/pg_search-pg13 ~/rpmbuild/SOURCES/pg_search_13;
rm -rf ~/rpmbuild/SOURCES/pg_search_12; cp -r ~/paradedb/target/release/pg_search-pg12 ~/rpmbuild/SOURCES/pg_search_12;

# packaging with rpmbuild
cd ~/rpmbuild/SPECS && make pg_search
rpmbuild --without debuginfo --define "pgmajorversion 16" -ba ~/rpmbuild/SPECS/pg_search.spec
rpmbuild --without debuginfo --define "pgmajorversion 15" -ba ~/rpmbuild/SPECS/pg_search.spec
rpmbuild --without debuginfo --define "pgmajorversion 14" -ba ~/rpmbuild/SPECS/pg_search.spec
rpmbuild --without debuginfo --define "pgmajorversion 13" -ba ~/rpmbuild/SPECS/pg_search.spec
rpmbuild --without debuginfo --define "pgmajorversion 12" -ba ~/rpmbuild/SPECS/pg_search.spec

pg_lakehouse

cd ~/paradedb/pg_lakehouse/
cargo update

# build pg_lakehouse
pg16 build
pg15 build

# move to rpmbuild SOURCES dir
rm -rf ~/rpmbuild/SOURCES/pg_lakehouse_16;  cp -r ~/paradedb/target/release/pg_lakehouse-pg16   ~/rpmbuild/SOURCES/pg_lakehouse_16;
rm -rf ~/rpmbuild/SOURCES/pg_lakehouse_15;  cp -r ~/paradedb/target/release/pg_lakehouse-pg15   ~/rpmbuild/SOURCES/pg_lakehouse_15;

# packaging with rpmbuild
cd ~/rpmbuild/SPECS && make pg_lakehouse
rpmbuild --without debuginfo --define "pgmajorversion 16" -ba ~/rpmbuild/SPECS/pg_lakehouse.spec
rpmbuild --without debuginfo --define "pgmajorversion 15" -ba ~/rpmbuild/SPECS/pg_lakehouse.spec

6 - Build DuckDB FDW

How to build DuckDB FDW extension and its dependencies libduckdb.so

DuckDB FDW depends on the libduckdb.so dynamic library. On Ubuntu and Debian, you can directly use the precompiled binaries provided by DuckDB.

On EL-based operating systems, you need to compile libduckdb from scratch and use this version to compile duckdb_fdw.

Pigsty’s github.com/pgsty/pgsql-rpm repository offers pre-built templates:

DuckDB has provided the libduckdb-src.zip source code, making it convenient for building.

Build Notes

You can only build this on EL8 & EL9 (EL7 not supported), to build duckdb_fdw, you also need to build libduckdb from source.

To build libduckdb-src, use the same building args as building duckdb_fdw:

clang++ -c -fPIC -std=c++11 -D_GLIBCXX_USE_CXX11_ABI=0 duckdb.cpp -o duckdb.o
clang++ -shared -o libduckdb.so *.o

To build libduckdb:

cd ~/rpmbuild/SPECS && make libduckdb

To build duckdb_fdw:

cd ~/rpmbuild/SPECS && make duckdb_fdw

7 - Build Parquet S3 FDW

Build Parquet S3 FDW extension alone with its deps: libarrow, libparquet, 以及 libaws-cpp

There are two major deps for parquet_s3_fdw: arrow 与 awssdk

Build arrow

Clone arrow repo and build it with cmake:

cd ~ ; git clone [email protected]:apache/arrow.git;
mkdir -p ~/arrow/cpp/release; cd ~/arrow/cpp/release;
cmake .. -DARROW_PARQUET=ON -DARROW_S3=ON; make -j8
sudo make install

Build libaws

There are many drivers in libaws-cpp, but we only need two: core and s3:

# install building deps
sudo yum install libcurl-devel openssl-devel libuuid-devel pulseaudio-libs-devel
# sudo apt-get install libcurl4-openssl-dev libssl-dev uuid-dev libpulse-dev # debian/ubuntu

# clone libaws repo (very big!)
cd ~; git clone --recurse-submodules [email protected]:aws/aws-sdk-cpp.git

mkdir -p ~/aws-sdk-cpp/release; cd ~/aws-sdk-cpp/release;
cmake .. -DBUILD_ONLY="s3"; make -j20
sudo make install

build libarrow-s3

Collect the generated .so files, then package them into an RPM / DEB package:

mkdir -p ~/libarrow-s3
cp -d ~/arrow/cpp/release/release/libarrow.so*                                     ~/libarrow-s3/
cp -d ~/arrow/cpp/release/release/libparquet.so*                                   ~/libarrow-s3/
cp -f ~/aws-sdk-cpp/release/generated/src/aws-cpp-sdk-s3/libaws-cpp-sdk-s3.so      ~/libarrow-s3/
cp -f ~/aws-sdk-cpp/release/src/aws-cpp-sdk-core/libaws-cpp-sdk-core.so            ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/lib/libaws-c-event-stream.so*                          ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/lib/libs2n.so*                                         ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/crt/aws-crt-cpp/libaws-crt-cpp.so                        ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/crt/aws-crt-cpp/crt/aws-c-common/libaws-c-common.so*     ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/crt/aws-crt-cpp/crt/aws-checksums/libaws-checksums.so*   ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/crt/aws-crt-cpp/crt/aws-c-io/libaws-c-io.so*             ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/crt/aws-crt-cpp/crt/aws-c-mqtt/libaws-c-mqtt.so*         ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/crt/aws-crt-cpp/crt/aws-c-cal/libaws-c-cal.so*           ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/crt/aws-crt-cpp/crt/aws-checksums/libaws-checksums.so*   ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/crt/aws-crt-cpp/crt/aws-c-s3/libaws-c-s3.so*             ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/crt/aws-crt-cpp/crt/aws-c-common/libaws-c-common.so*     ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/crt/aws-crt-cpp/crt/aws-c-http/libaws-c-http.so*         ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/crt/aws-crt-cpp/crt/aws-c-sdkutils/libaws-c-sdkutils.so* ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/crt/aws-crt-cpp/crt/aws-c-auth/libaws-c-auth.so*         ~/libarrow-s3/
cp -d ~/aws-sdk-cpp/release/crt/aws-crt-cpp/crt/aws-c-compression/libaws-c-compression.so* ~/libarrow-s3/

Remove empty RPATH from generated so files (EL system), using patchelf binary:

cd ~/libarrow-s3/
patchelf --remove-rpath libarrow.so.1800.0.0
patchelf --remove-rpath libparquet.so.1800.0.0
patchelf --remove-rpath libaws-cpp-sdk-core.so
patchelf --remove-rpath libaws-cpp-sdk-s3.so

And finally package these so files into a libarrow-s3 package:

cd ~/rpmbuild/SPECS
rpmbuild -ba ~/rpmbuild/SPECS/libarrow-s3.spec
sudo rpm -ivh ~/rpmbuild/RPMS/x86_64/libarrow-s3-17.0.0-1PIGSTY.*