mirror of https://github.com/leafspark/AutoGGUF
162 lines
6.0 KiB
Markdown
162 lines
6.0 KiB
Markdown
|
||
|
||
# AutoGGUF - automated GGUF model quantizer
|
||
|
||
<!-- Project Status -->
|
||
[](https://github.com/leafspark/AutoGGUF/releases)
|
||
[](https://github.com/leafspark/AutoGGUF/commits)
|
||
[]()
|
||
|
||
<!-- Project Info -->
|
||
[](https://github.com/ggerganov/llama.cpp)
|
||
[]()
|
||
[](https://github.com/leafspark/AutoGGUF/blob/main/LICENSE)
|
||
|
||
|
||
<!-- Repository Stats -->
|
||
|
||
|
||
|
||
|
||
|
||
|
||
<!-- Contribution -->
|
||
[](https://github.com/leafspark/AutoGGUF/issues)
|
||
[](https://github.com/psf/black)
|
||
[](https://github.com/leafspark/AutoGGUF/pulls)
|
||
|
||
The most comprehensive GUI tool for GGUF model quantization. Stop wrestling with command lines - quantize, merge, and optimize your models with just a few clicks.
|
||
|
||
## Features
|
||
|
||
- 📩 Download and manage llama.cpp backends
|
||
- 🗃️ Select and quantize GGUF models
|
||
- 📐 Configure quantization parameters
|
||
- 💻 Monitor system resources during quantization
|
||
- ⏳ Parallel quantization + imatrix generation
|
||
- 🎉 LoRA conversion and merging
|
||
- 📁 Preset saving and loading
|
||
- 8️⃣ AutoFP8 quantization
|
||
- 🪓 GGUF splitting and merging
|
||
- 🌐 HTTP API for automated monitoring
|
||
|
||
## Why AutoGGUF?
|
||
- Fast: Saves time on manual configuration
|
||
- Simple: Clean UI, no terminal needed
|
||
- Powerful: Handles models up to infinite size, only limited by your RAM
|
||
- Resource-aware: Optimized memory management and efficient UI library
|
||
|
||
|
||
|
||
## Quick Start
|
||
|
||
### Cross-platform
|
||
1. `git clone https://github.com/leafspark/AutoGGUF`
|
||
2. `cd AutoGGUF`
|
||
3. Install dependencies:
|
||
```
|
||
pip install -r requirements.txt
|
||
```
|
||
4. Run the application:
|
||
```
|
||
python src/main.py
|
||
```
|
||
or use the `run.bat` script.
|
||
|
||
macOS and Ubuntu builds are provided with GitHub Actions, you may download the binaries in the releases section.
|
||
|
||
### Windows (for the impatient)
|
||
Standard builds:
|
||
1. Download the latest release
|
||
2. Extract all files to a folder
|
||
3. Run `AutoGGUF-x64.exe`
|
||
4. Any necessary folders will be automatically created
|
||
|
||
Setup builds:
|
||
1. Download setup variant of latest release
|
||
2. Extract all files to a folder
|
||
3. Run the setup program
|
||
4. The .GGUF extension will be registered with the program automatically
|
||
5. Run the program from the Start Menu or desktop shortcuts
|
||
|
||
After launching the program, you may access its local server at port 7001 (set `AUTOGGUF_SERVER` to "enabled" first)
|
||
|
||
### Verifying Releases
|
||
|
||
#### Linux/macOS:
|
||
```bash
|
||
gpg --import AutoGGUF-v1.5.0-prerel.asc
|
||
gpg --verify AutoGGUF-v1.9.1-Windows-avx2.zip.sig AutoGGUF-v1.9.1-Windows-avx2.zip
|
||
sha256sum -c AutoGGUF-v1.9.1.sha256
|
||
```
|
||
|
||
#### Windows (PowerShell):
|
||
```powershell
|
||
# Import the public key
|
||
gpg --import AutoGGUF-v1.5.0-prerel.asc
|
||
|
||
# Verify the signature
|
||
gpg --verify AutoGGUF-v1.9.1-Windows-avx2.zip.sig AutoGGUF-v1.9.1-Windows-avx2.zip
|
||
|
||
# Check SHA256
|
||
$fileHash = (Get-FileHash -Algorithm SHA256 AutoGGUF-v1.9.1-Windows-avx2.zip).Hash.ToLower()
|
||
$storedHash = (Get-Content AutoGGUF-v1.9.1.sha256 | Select-String AutoGGUF-v1.9.1-Windows-avx2.zip).Line.Split()[0]
|
||
if ($fileHash -eq $storedHash) { "SHA256 Match" } else { "SHA256 Mismatch" }
|
||
```
|
||
|
||
Release keys are identical to ones used for commiting.
|
||
|
||
## Building
|
||
|
||
### Cross-platform
|
||
```bash
|
||
pip install -U pyinstaller
|
||
./build.sh RELEASE | DEV
|
||
cd build/<type>/dist/
|
||
./AutoGGUF
|
||
```
|
||
|
||
### Windows
|
||
```bash
|
||
pip install -U pyinstaller
|
||
build RELEASE | DEV
|
||
```
|
||
Find the executable in `build/<type>/dist/AutoGGUF.exe`.
|
||
|
||
You can also use Nuitka, which may result in a slower build but a faster output executable:
|
||
```bash
|
||
build_optimized RELEASE | DEV
|
||
```
|
||
|
||
## Localizations
|
||
|
||
View the list of supported languages at [AutoGGUF/wiki/Installation#configuration](https://github.com/leafspark/AutoGGUF/wiki/Installation#configuration) (LLM translated, except for English).
|
||
|
||
More languages will be updated as soon as possible!
|
||
|
||
To use a specific language, set the `AUTOGGUF_LANGUAGE` environment variable to one of the listed language codes (note: some languages may not be fully supported yet, those will fall back to English).
|
||
|
||
## Issues
|
||
|
||
- Some inconsistent logging
|
||
|
||
## Planned Features
|
||
|
||
- Time estimation for quantization
|
||
- Quantization file size estimate
|
||
- Perplexity testing
|
||
- bitsandbytes (coming soon)
|
||
|
||
## Support
|
||
|
||
- SSL module cannot be found error: Install OpenSSL or run from source using `python src/main.py` with the `run.bat` script (`pip install requests`)
|
||
- Check out the [Wiki](https://github.com/leafspark/AutoGGUF/wiki) for advanced usage
|
||
|
||
## Contributing
|
||
|
||
Fork the repo, make your changes, and ensure you have the latest commits when merging. Include a changelog of new features in your pull request description. Read `CONTRIBUTING.md` for more information.
|
||
|
||
## Stargazers
|
||
|
||
[](https://star-history.com/#leafspark/AutoGGUF&Date)
|