#Large stack of books how to#
#Large stack of books code#
#Large stack of books download#

Indicates whether a line ends with a terminal punctuation mark. Rps_lines_ending_with_terminal_punctution_mark The number of words in the content after normalisation.

This measures the diversity of the content and is computed using sum(-x / total * log(x / total)) where the sum is taken over counts of unique words in the normalised content. The entropy of the unigram distribution of the content. Calculated based on the normalised content. This is also known as the degeneracy of a text sample. The fraction of unique words in the content. The ratio of symbols to words in the content. Stop words are obtained from the stopwords-json repo. The ratio between the number of stop words and the number of words in the document. The mean length of words in the content after normalisation. The ratio between the number of occurrences of 'lorem ipsum' and the number of characters in the content after normalisation. The fraction of words that contain no alphabetical character. The fraction of lines that end with an ellipsis, where an ellipsis is defined as either '.' or '…'. The fraction of words in the content that only consist of uppercase letters. In $' and the number of characters in the raw text.

#Large stack of books download#

The most recent list of blacklisted urls from the UT1 blacklist.Īs a first step, download the english wikipedia reference classifier This includes building qualityĬlassifiers, training bag-of-ngram generative models for importance weight computation, fetching the list of bad words This part of the pipeline creates the artifacts that are used in the subsequent steps. The pipeline is composed of three steps, namely 1) preparing artifacts, 2) computing quality signals, and 3) Have a docker and apptainer installation. You can run the steps of the pipeline without any containerized environment. To run with docker, build the docker image usingĪlso, make sure you have s5cmd installed and your S3 profile configured so that you can pull data from an S3 bucket. These will be used throughout the pipeline. configs/nf and configure the environment variables. The number of documents and tokens for the annotated and deduplicated head_middle part of the dataset is shown in theĮnglish, German, French, Italian, Spanish Setup ConfigurationĬopy the file configs/rp_v2.0.conf to e.g. Document and Token Counts for the Annotated and deduplicated head_middle part of the dataset That additionally come with quality signals, and 20B documents that are deduplicated.

Out of these, there are 30B documents in the corpus The dataset includes over 100B textĭocuments coming from 84 CommonCrawl snapshots and processed using RedPajama-V2 is an open dataset for training large language models. RedPajama-1T dataset, please refer to the rp_v1 branch in this repo. For more information on the dataset, check out ourīlog post.

#Large stack of books code#

This repository contains the code for the RedPajama-V2 dataset. In fact, if you’ve hung around this blog, you’ve probably seen most of the books below mentioned before.RedPajama-Data-v2: an Open Dataset with 30 Trillion Tokens for Training Large Language Models Many of my favorite authors are still favorites. My tastes, overall, haven’t changed a lot. Keeping reading journals to track the books I read, the books I want to read, and the books I love is another way I try to balance reading old and new books.

#Large stack of books how to#

I’m still trying to figure out how to balance the older books I want to read with the new and shiny books I can’t resist.Īudiobooks are one way I handle that–and a major change to my reading this past year, as I started listening more and more. Of course, my TBR only continues to grow. It was a whole new way to enjoy my favorite hobby, and to share what had so long been a solitary pursuit with others.Įventually, I started reading mostly new releases and ventured into advanced reader copies–mostly because it’s fun to discuss the newest buzzy books with others who are also just discovering them. I started paying more attention to the book community, bookish news, and the books that others were loving. Starting this blog a few years ago changed my reading again.