generated from ztimson/template
Migrated to kiwix-serve
This commit is contained in:
159
README.md
159
README.md
@@ -1,4 +1,3 @@
|
||||
<!-- Header -->
|
||||
<div id="top" align="center">
|
||||
<br />
|
||||
|
||||
@@ -9,7 +8,7 @@
|
||||
### Zim Utils
|
||||
|
||||
<!-- Description -->
|
||||
Native, dependency-light ZIM archive reader/searcher and Kiwix catalog downloader for Node.js
|
||||
Node.js wrapper around [kiwix-tools](https://github.com/kiwix/kiwix-tools) for managing a local ZIM library, serving content, and searching both locally and against the Kiwix catalog
|
||||
|
||||
<!-- Repo badges -->
|
||||
[](https://git.zakscode.com/ztimson/zim-utils/tags)
|
||||
@@ -36,24 +35,23 @@ Native, dependency-light ZIM archive reader/searcher and Kiwix catalog downloade
|
||||
- [Setup](#setup)
|
||||
- [Production](#production)
|
||||
- [Usage](#usage)
|
||||
- [ZimManager](#zimmanager)
|
||||
- [ZimReader](#zimreader)
|
||||
- [KiwixServer](#kiwixserver)
|
||||
- [Catalog](#catalog)
|
||||
- [License](#license)
|
||||
|
||||
## About
|
||||
|
||||
`@ztimson/zim-utils` is a native, dependency-light toolkit for working with [ZIM](https://wiki.openzim.org/wiki/ZIM_file_format) archives and the [Kiwix](https://kiwix.org/) catalog in Node.js
|
||||
`@ztimson/zim-utils` manages a local [ZIM](https://wiki.openzim.org/wiki/ZIM_file_format) library by driving the official [kiwix-tools](https://github.com/kiwix/kiwix-tools) binaries (`kiwix-serve`, `kiwix-manage`) as child processes, rather than re-implementing ZIM reading from scratch. This means content serving, indexing, and fulltext search all come straight from Kiwix's own battle-tested implementation.
|
||||
|
||||
It comes with the following helpers:
|
||||
It comes with:
|
||||
|
||||
- **`ZimManager`** — Local library manager: listing, updating/downloading, and running searches accross the entire library
|
||||
- **`ZimReader`** — A `.zim` file reader for reading pages, metadata and running searches
|
||||
- **Catalog** (`zimCatalog`/`zimCatalogInfo`) — Helper functions to search the Kiwix OPDS catalog
|
||||
- **`KiwixServer`** — Owns a directory of `.zim` files: builds/rebuilds `library.xml`, starts/stops `kiwix-serve`, lists the local library, runs fulltext search, and resolves content URLs/bytes for any archive+page
|
||||
- **Catalog** (`zimCatalog`/`zimCatalogInfo`) — Helper functions to search the remote Kiwix OPDS catalog, for discovering and downloading new ZIMs
|
||||
|
||||
### Built With
|
||||
[](https://nodejs.org/)
|
||||
[](https://javascript.com/)
|
||||
[](https://github.com/kiwix/kiwix-tools)
|
||||
|
||||
## Setup
|
||||
|
||||
@@ -68,84 +66,121 @@ It comes with the following helpers:
|
||||
- [Node.js](https://nodejs.org/en/download)
|
||||
|
||||
#### Instructions
|
||||
1. Install the dependencies: `npm install`
|
||||
2. Install the package: `npm install @ztimson/zim-utils`
|
||||
1. Install the package: `npm install @ztimson/zim-utils`
|
||||
2. On first install, `kiwix-serve`/`kiwix-manage` binaries are fetched into `./bin` (or wherever `binDir` points) - no separate Kiwix install needed.
|
||||
|
||||
</details>
|
||||
|
||||
## Usage
|
||||
|
||||
### ZimManager
|
||||
### KiwixServer
|
||||
|
||||
`ZimManager` owns a directory of `.zim` files and handles everything from downloading to cross-archive search.
|
||||
`KiwixServer` owns a directory of `.zim` files and manages the whole `kiwix-serve` lifecycle for you.
|
||||
|
||||
```js
|
||||
import {ZimManager} from '@ztimson/zim-utils';
|
||||
import {KiwixServer} from '@ztimson/zim-utils';
|
||||
|
||||
const manager = new ZimManager('./zims'); // optional 2nd arg: custom catalog URL
|
||||
|
||||
// Search the Kiwix catalog & download the top hit
|
||||
const [entry] = await manager.catalog('wikipedia,medicine');
|
||||
await manager.download(entry.href);
|
||||
|
||||
// List local archives with their parsed metadata
|
||||
const local = await manager.list();
|
||||
// [{file: './zims/wikipedia_en_medicine.zim', meta: {name, date, title}}, ...]
|
||||
|
||||
// Check a single file for updates without downloading
|
||||
const status = await manager.isOutdated(local[0].file);
|
||||
|
||||
// Update every local ZIM that has a newer catalog version
|
||||
await manager.updateAll({force: false});
|
||||
|
||||
// Fuzzy-search titles across ALL local archives at once
|
||||
const hits = await manager.search('diabetes treatment', {limit: 10});
|
||||
|
||||
// Open a reader by file path OR by catalog name
|
||||
const reader = await manager.open('wikipedia_en_medicine');
|
||||
const page = await reader.readPage('A/Diabetes');
|
||||
await reader.close();
|
||||
const server = new KiwixServer('./zims'); // optional 2nd arg: {port, host, binDir}
|
||||
await server.start(); // rebuilds library.xml, spawns kiwix-serve, waits until ready
|
||||
await server.reload(); // pick up newly added/removed .zim files, no downtime
|
||||
await server.stop(); // Kill server
|
||||
```
|
||||
|
||||
### ZimReader
|
||||
Everything `ZimManager` does to a single archive is just a thin wrapper around `ZimReader`. Use it directly when you don't need a whole managed library:
|
||||
#### List Local ZIMs
|
||||
```js
|
||||
const local = await server.list();
|
||||
[
|
||||
{
|
||||
id: '37a99758-43a0-6ba3-cd54-1af556369eee',
|
||||
title: 'FOSS cooking',
|
||||
updated: 2026-05-06T00:00:00.000Z,
|
||||
summary: 'Making cooking fast, easy, foss',
|
||||
language: 'eng',
|
||||
name: 'foss.cooking_en_all',
|
||||
category: 'other',
|
||||
tags: ['_category:other', '_ftindex:yes', 'preppers,food', '_pictures:yes', '_videos:yes', '_details:yes'],
|
||||
mediaCount: 154,
|
||||
author: '-',
|
||||
publisher: 'openZIM',
|
||||
articleCount: 719,
|
||||
sizeMb: 23.1,
|
||||
href: 'foss.cooking_en_all_2026-05',
|
||||
icon: undefined,
|
||||
viewer: 'http://127.0.0.1:51992/content/foss.cooking_en_all_2026-05',
|
||||
},
|
||||
// ...
|
||||
]
|
||||
```
|
||||
|
||||
#### Search Local ZIMs
|
||||
|
||||
```js
|
||||
import {ZimReader} from '@ztimson/zim-utils';
|
||||
// Fulltext search across every local archive: xapian prefilter, then fuzzy re-ranked
|
||||
const hits = await server.search('chocolate', 5);
|
||||
[
|
||||
{
|
||||
id: '37a99758-43a0-6ba3-cd54-1af556369eee',
|
||||
title: 'Chocolate Chip Cookies',
|
||||
page: 'foss.cooking/recipe/mfed3/chocolate-chip-cookies',
|
||||
name: 'foss.cooking_en_all',
|
||||
publisher: 'openZIM',
|
||||
href: 'foss.cooking_en_all_2026-05/foss.cooking/recipe/mfed3/chocolate-chip-cookies',
|
||||
icon: undefined,
|
||||
viewer: 'http://127.0.0.1:51992/content/foss.cooking_en_all_2026-05/foss.cooking/recipe/mfed3/chocolate-chip-cookies',
|
||||
summary: '...cream scooper or spoon to make uniform balls of cookie dough...',
|
||||
score: 0.67,
|
||||
},
|
||||
// ...
|
||||
]
|
||||
```
|
||||
|
||||
const reader = await new ZimReader('./zims/wikipedia_en_medicine.zim').open();
|
||||
#### View ZIM Content
|
||||
|
||||
// Metadata (what manager.#readMeta / isOutdated rely on)
|
||||
const name = await reader.metadata('Name');
|
||||
const date = await reader.metadata('Date');
|
||||
const title = await reader.metadata('Title');
|
||||
```js
|
||||
// Build a content URL from any href (list()/search() output, or a full viewer URL) without hitting the network
|
||||
const url = server.fetch(hits[0].href);
|
||||
'http://127.0.0.1:51992/content/foss.cooking_en_all_2026-05/foss.cooking/recipe/mfed3/chocolate-chip-cookies'
|
||||
|
||||
// Landing page
|
||||
const home = await reader.mainPage();
|
||||
|
||||
// Direct page lookup by URL
|
||||
const page = await reader.readPage('A/Diabetes');
|
||||
console.log(page.mimetype, page.data.toString('utf8'));
|
||||
|
||||
// Fuzzy title search within just this archive (what manager.search fans out over)
|
||||
const results = await reader.search('diabetes,insulin', {limit: 20, htmlOnly: true});
|
||||
|
||||
await reader.close();
|
||||
// Or fetch the bytes directly, proxied straight from kiwix-serve
|
||||
const {mimetype, data} = await server.raw(hits[0].href);
|
||||
{mimetype: 'text/html; charset=utf-8', data: <Buffer ...>}
|
||||
```
|
||||
|
||||
### Catalog
|
||||
`ZimManager.catalog()` and its update checks are backed directly by these two functions:
|
||||
|
||||
```js
|
||||
import {zimCatalog, zimCatalogInfo, CATALOG_URL} from '@ztimson/zim-utils';
|
||||
|
||||
// Ranked search across the Kiwix catalog (comma-separated terms, like ZimReader.search)
|
||||
const results = await zimCatalog('history,rome', {lang: 'eng', count: 20, url: CATALOG_URL});
|
||||
// Search the remote Kiwix catalog to discover new ZIMs
|
||||
const search = await zimCatalog('knots', {lang: 'eng', count: 20, url: CATALOG_URL});
|
||||
// OR Exact lookup by catalog `name`, useful for checking if a local copy is outdated
|
||||
const entry = await zimCatalogInfo('wikipedia_en_knots');
|
||||
|
||||
// Exact lookup by catalog `name`, used to check if a local copy is outdated
|
||||
const entry = await zimCatalogInfo('wikipedia_en_medicine');
|
||||
[
|
||||
{
|
||||
id: 'urn:uuid:3a4fe0d0-0bd0-7583-ada8-c52d173ae44d',
|
||||
title: 'Knots by Wikipedia',
|
||||
updated: 2026-07-20T00:00:00.000Z,
|
||||
summary: 'A subset of Wikipedia encyclopedia dedicated to knots',
|
||||
language: 'eng',
|
||||
name: 'wikipedia_en_knots',
|
||||
category: 'wikipedia',
|
||||
tags: ['wikipedia', '_category:wikipedia', '_pictures:yes', '_videos:no', '_details:yes', '_ftindex:yes'],
|
||||
mediaCount: 3501,
|
||||
author: 'Wikipedia',
|
||||
publisher: 'openZIM',
|
||||
articleCount: 1730,
|
||||
sizeMb: 17.6,
|
||||
href: 'https://lb.download.kiwix.org/zim/wikipedia/wikipedia_en_knots_maxi_2026-07.zim.meta4',
|
||||
icon: 'https://library.kiwix.org/catalog/v2/illustration/3a4fe0d0-0bd0-7583-ada8-c52d173ae44d/?size=48',
|
||||
viewer: 'https://library.kiwix.org/viewer#wikipedia_en_knots',
|
||||
},
|
||||
// ...
|
||||
]
|
||||
|
||||
// Download catalog ZIM if out of date (unless forced) and reload automatically
|
||||
await mgr.download(search[0].href, force);
|
||||
```
|
||||
|
||||
## License
|
||||
Copyright © 2026 Zakary Timson | Available under MIT Licensing
|
||||
|
||||
|
||||
Reference in New Issue
Block a user