OSCAR

All

16 repositories

documentation
Public
Apache License 2.0
•4•5•9•0•Updated Nov 13, 2024Nov 13, 2024
oscar-statistics
Public
Compute statistics for OSCAR Monthly releases
Rust
•
Apache License 2.0
•0•2•0•0•Updated Sep 2, 2024Sep 2, 2024
oscar-io
Public
Readers/Writers for OSCAR Corpus
Rust
•
Apache License 2.0
•1•0•2•1•Updated Apr 16, 2024Apr 16, 2024
oscar-tools
Public
The original tooling for the OSCAR corpus rewritten in Rust
Rust
•
Apache License 2.0
•3•5•9•6•Updated Dec 19, 2023Dec 19, 2023
ungoliant
Public
🕷️ The pipeline for the OSCAR corpus
nlp crawler corpus-linguistics fasttext oscar commoncrawl common-crawl language-classification
Rust
•
Apache License 2.0
•15•166•18•13•Updated Dec 18, 2023Dec 18, 2023
ut1-rs
Public
ut1-blocklist rust library
Rust
•
MIT License
•0•1•1•0•Updated Nov 9, 2023Nov 9, 2023
oscar-website
Public
The website of the Oscar Project
nlp website machine-learning language-model common-crawl hugo
TeX
•
Apache License 2.0
•14•11•2•0•Updated Nov 9, 2023Nov 9, 2023
.github
Public
1•0•0•1•Updated Aug 8, 2023Aug 8, 2023
OSCAR-CommonCrawl-Collab
Public
Jupyter Notebook
•0•2•2•0•Updated Mar 12, 2023Mar 12, 2023
download_oscar
Public
Downloading all files of a language from the OSCAR (Open Super-large Crawled Aggregated coRpus)
Python
•
MIT License
•2•10•0•1•Updated Feb 28, 2023Feb 28, 2023
data-hub
Public
Collab around OSCAR: Data soucing..
Python
•0•1•1•0•Updated Feb 9, 2023Feb 9, 2023
discussions
Public
0•0•0•0•Updated Jan 18, 2023Jan 18, 2023
oscar-blocklists
Public
A compilation of multilingual URL blocklist
Creative Commons Zero v1.0 Universal
•1•2•0•0•Updated Dec 21, 2022Dec 21, 2022
corpus
Public
corpus issues.
Apache License 2.0
•0•5•20•0•Updated Oct 18, 2021Oct 18, 2021
oscar-tools-go
Public archive
A tooling for the OSCAR corpus
Go
•
Apache License 2.0
•0•1•0•0•Updated Jun 25, 2021Jun 25, 2021
goclassy
Public archive
An asynchronous concurrent pipeline for classifying Common Crawl based on fastText's pipeline.
nlp corpus-linguistics fasttext common-crawl language-classification
Go
•
Apache License 2.0
•6•86•0•0•Updated Apr 21, 2021Apr 21, 2021