User:Connel MacKenzie/Gutenberg

Definition from Wiktionary, the free dictionary
Jump to: navigation, search

Project Gutenberg[edit]

So, I downloaded all of Project Gutenberg.

Yes, I had to buy a new external hard drive first.

The raw frequency count is at Wiktionary:Frequency_lists. That lists words in their rank order. Unlike other possible rankings by "headword" this lists the exact inflection or conjugation's rank...much like Wiktionary has separate entries for each form of a word.

The following lists indicate words that were undefined in Wiktionary, as of whichever XML dump backup date this was re-run. Since the backup is done monthly, you should not expect to see this updated more than monthly. I will try to add new "runs" toward the top of this page, and keep older runs below.

The "rank" column in these lists reflect their relative rank amongst the words that were undefined in Wiktionary as of 2 October 2005.

Added filter for most common entries that don't meet WT:CFI. --Connel MacKenzie T C 17:46, 27 May 2006 (UTC)

As of the 12/14/2008 XML dump, this garners no new entries at all, so is hereby discontinued. --Connel MacKenzie 15:34, 15 December 2008 (UTC)

2008-11-11 XML dump[edit]

This information on this page is from the 2008-11-11 database dump so it is not exactly up to date.

26053 (53544) files, 1,803,079,588 words, 69208.1 average words/file.

2008-05-20 XML dump[edit]

This information on this page is from the 2008-05-20 database dump so it is not exactly up to date.

23887 (48828) files, 1,670,451,881 words, 69931.4 average words/file.

2008-03-16 XML dump[edit]

This information on this page is from the 2008-03-16 database dump so it is not exactly up to date.

23334 (47595) files, 1,638,680,591 words, 70227.2 average words/file.

2008-03-03 XML dump[edit]

This information on this page is from the 2008-03-03 database dump so it is not exactly up to date.

23247 (47419) files, 1,632,724,121 words, 70233.8 average words/file.

2008-02-12 XML dump[edit]

This information on this page is from the 2008-02-12 database dump so it is not exactly up to date.

23104 (47154) files, 1,624,715,368 words, 70321.8 average words/file.

2008-01-16 XML dump[edit]

This information on this page is from the 2008-01-16 database dump so it is not exactly up to date.

22834 (46626) files, 1,609,998,963 words, 70508.8 average words/file.

2008-01-02 XML dump[edit]

This information on this page is from the 2008-01-02 database dump so it is not exactly up to date.

22717 (46355) files, 1,603,759,837 words, 70597.3 average words/file.

2007-11-20 XML dump[edit]

This information on this page is from the 2007-11-20 database dump so it is not exactly up to date.

22292 (45431) files, 1,582,466,151 words, 70988.1 average words/file.

2007-10-16 XML dump[edit]

This information on this page is from the 2007-10-16 database dump so it is not exactly up to date.

21827 (44651) files, 1,560,487,948 words, 71493.5 average words/file.

2007-09-29 XML dump[edit]

This information on this page is from the 2007-09-29 database dump so it is not exactly up to date.

21683 (44339) files, 1,554,621,253 words, 71697.7 average words/file.

2007-09-13 XML dump[edit]

This information on this page is from the 2007-09-13 database dump so it is not exactly up to date.

21496 (43955) files, 1,544,157,117 words, 71834.6 average words/file.

2007-08-29 XML dump[edit]

This information on this page is from the 2007-08-29 database dump so it is not exactly up to date.

21348 (43643) files, 1,535,706,380 words, 71936.8 average words/file.

2007-08-13 XML dump[edit]

Monday the 13th.
This information on this page is from the 2007-08-13 database dump so it is not exactly up to date.

21280 (43509) files, 1,531,671,679 words, 71977.1 average words/file.

2007-07-20 XML dump[edit]

This information on this page is from the 2007-07-20 database dump so it is not exactly up to date.

21070 (43136) files, 1,518,949,920 words, 72090.6 average words/file.

2007-06-28 XML dump[edit]

This information on this page is from the 2007-06-28 database dump so it is not exactly up to date.

20941 (42857) files, 1,509,794,423 words, 72097.5 average words/file.

2007-05-25 XML dump[edit]

This information on this page is from the 2007-05-25 database dump so it is not exactly up to date.

20594 (42256) files, 1,487,309,991 words, 72220.5 average words/file.

2007-05-13 XML dump[edit]

This information on this page is from the 2007-05-13 database dump so it is not exactly up to date.

20441 (42004) files, 1,474,994,456 words, 72158.6 average words/file.

2007-04-29 XML dump[edit]

This information on this page is from the 2007-04-29 database dump so it is not exactly up to date.

20193 (41566) files, 1,455,619,521 words, 72085.4 average words/file.

2007-04-19 XML dump[edit]

This information on this page is from the 2007-04-19 database dump so it is not exactly up to date.

20193 (41566) files, 1,455,619,521 words, 72085.4 average words/file.

With XML dumps increasing frequency, I'll do 1,000 only infrequently now.

4/08/2007 XML dump[edit]

This information on this page is from the 2007-4-08 database dump so it is not exactly up to date.

20015 (41217) files, 1,441,796,132 words, 72035.8 average words/file.

3/28/2007 XML dump[edit]

This information on this page is from the 2007-3-28 database dump so it is not exactly up to date.

20014 (41216) files, 1,441,750,666 words, 72037.1 average words/file.

2/25/2007 XML dump[edit]

This information on this page is from the 2007-2-25 database dump so it is not exactly up to date.

19795 (40765) files, 1,427,148,930 words, 72096.4 average words/file.

2/09/2007 XML dump[edit]

This information on this page is from the 2007-2-09 database dump so it is not exactly up to date.

19626 (40421) files, 1,416,275,788 words, 72163.2 average words/file.

1/26/2007 XML dump[edit]

This information on this page is from the 2007-1-26 database dump so it is not exactly up to date.

19625 (40419) files, 1,416,250,763 words, 72165.6 average words/file.

12/24/2006 XML dump[edit]

This information on this page is from the 2006-12-24 database dump so it is not exactly up to date.

19369 (39897) files, 1,400,560,621 words, 72309.4 average words/file.

12/02/2006 XML dump[edit]

This information on this page is from the 2006-12-01 database dump so it is not exactly up to date.

19155 (39460) files, 1,386,400,434 words, 72378.0 average words/file.

11/03/2006 XML dump[edit]

This information on this page is from the 2006-11-03 database dump so it is not exactly up to date.

18987 (39139) files, 1,375,932,670 words, 72467.1 average words/file.

10/17/2006 XML dump:[edit]

This information on this page is from the 10/17/2006 database dump so it is not exactly up to date.

18616 (38463) files, 1,354,350,877 words, 72752.0 average words/file.

9/27/2006 XML dump:[edit]

This information on this page is from the 9/27/2006 database dump so it is not exactly up to date.

18616 (38463) files, 1,354,350,877 words, 72752.0 average words/file.

9/22/2006 XML dump:[edit]

This information on this page is from the 9/22/2006 database dump so it is not exactly up to date.

18490 (38250) files, 1,344,040,355 words, 72690.1 average words/file.

9/12/2006 XML dump:[edit]

This information on this page is from the 9/13/2006 database dump so it is not exactly up to date.

18490 (38250) files, 1,344,040,355 words, 72690.1 average words/file.

8/5/2006 XML dump:[edit]

This information on this page is from the 8/05/2006 database dump so it is not exactly up to date.

18320 (37892) files, 1,333,148,479 words, 72770.1 average words/file.

7/14/2006 XML dump:[edit]

This information on this page is from the 7/14/2006 database dump so it is not exactly up to date.

17871 (36999) files, 1,301,860,862 words, 72847.7 average words/file.

7/04/2006 XML dump:[edit]

This information on this page is from the 7/04/2006 database dump so it is not exactly up to date.

17871 (36999) files, 1,301,860,862 words, 72847.7 average words/file.

6/19/2006 XML dump:[edit]

This information on this page is from the 6/19/2006 database dump so it is not exactly up to date.

17871 (36999) files, 1,301,860,862 words, 72847.7 average words/file.

5/28/2006 XML dump:[edit]

This information on this page is from the 5/28/2006 database dump so it is not exactly up to date.

17650 (36540) files, 1,286,567,649 words, 72893.4 average words/file.

5/20/2006 XML dump:[edit]

This information on this page is from the 5/20/2006 database dump so it is not exactly up to date.

17650 (36540) files, 1,286,567,649 words, 72893.4 average words/file.

5/02/2006 XML dump:[edit]

This information on this page is from the 5/02/2006 database dump so it is not exactly up to date.

17376 (27739) files, 1,264,063,754 words, 72747.7 average words/file.

4/01/2006 XML dump:[edit]

This information on this page is from the 4/01/2006 database dump so it is not exactly up to date.

After re-parsing English texts only, excluding the many duplicates:

17376 (27739) files, 1,264,063,754 words, 72747.7 average words/file.

2/24/2006 XML dump:[edit]

This information on this page is from the 2/24/2006 database dump so it is not exactly up to date.

After re-parsing English texts only, excluding the many duplicates:

16173 (25114) files, 1,185,756,798 words, 73317.1 average words/file. (Hrm, I need to rsync with Gutenberg one of these days.)

1/29/2006 XML dump:[edit]

After re-parsing English texts only, excluding the many duplicates:

16173 (25114) files, 1,185,756,798 words, 73317.1 average words/file.

12/11/2005 XML dump:[edit]

After re-parsing English texts only, excluding the many duplicates:

16173 (25114) files, 1,185,756,798 words, 73317.1 average words/file.

11/27/2005 XML dump:[edit]

After re-parsing English texts only, excluding the many duplicates:

16173 (25114) files, 1,185,756,798 words, 73317.1 average words/file.

11/13/2005 XML dump:[edit]

After re-parsing English texts only, excluding the many duplicates:

16173 (25114) files, 1,185,756,798 words, 73317.1 average words/file.

11/05/2005 XML dump:[edit]

After re-parsing English texts only, excluding the many duplicates:

16173 files, 1,185,756,798 words, 73317.1 average words/file.

10/27/2005 XML dump:[edit]

After re-importing only the English language texts from www.gutenberg.org:

After the 10/20/2005 XML dump:[edit]

After 10/02/2005 XML dump:[edit]

After 8/14/2005 XML dump:[edit]

  • 20,000 books, top 5,000 words not defined in en.wiktionary by frequency:


Older stuff[edit]

This stuff was work-in-progress. I think I'll delete these in the next couple days.