# Indexing mostly-binary documents (.ppt) **URL:** \ **Category:** Ferret\ **Created:** [April 1, 2007, 3:18am UTC](https://www.ruby-forum.com/t/indexing-mostly-binary-documents-ppt/90763 "2007-04-01T03:18:27Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![johnnybutler7](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/8491ac/32.png) [@johnnybutler7](https://www.ruby-forum.com/u/johnnybutler7)\ **Post date:** [April 1, 2007, 3:18am UTC](https://www.ruby-forum.com/t/indexing-mostly-binary-documents-ppt/90763/1 "2007-04-01T03:18:27Z")
Here’s an interesting problem: In my app, we are indexing various types of documents, including microsoft powerpoint. Powerpoint documents are mostly binary, but have a bunch of text (all of the text in the document?) as well. My thinking is that the binary will never get searched for, and the proper text will be indexed and queried as expected, so the indexed binary will never affect results. Is this correct? Then my colleague mentioned that maybe the indexed garbage would affect the weighting of certain searches? I figure that weighting is only per-search so, same situation as above, only the proper terms will be calculated. What do you folks think? John ---
**Author:** ![johnnybutler7](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/8491ac/32.png) [@johnnybutler7](https://www.ruby-forum.com/u/johnnybutler7)\ **Post date:** [April 1, 2007, 11:48am UTC](https://www.ruby-forum.com/t/indexing-mostly-binary-documents-ppt/90763/2 "2007-04-01T11:48:05Z")
On Apr 1, 2007, at 3:09 AM, John B. wrote: > Here’s an interesting problem: In my app, we are indexing various > types of documents, including microsoft powerpoint. Powerpoint > documents are mostly binary, but have a bunch of text (all of the > text in the document?) as well. Are you serious? You’re adding raw, unprocessed PPT files to your index? Now this is just wrong. PPT files may contain all sorts of binary data, such as images and videos. I just had a look at the sample presentation that came with my Office installation. This file is 3.5MB in size with a (plain text) payload of less than 1KB. I’m sure there’s some tool available which converts PPT to plain text and I strongly recommend you go out and find it. Cheers, Andy ---
**Author:** ![johnnybutler7](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/8491ac/32.png) [@johnnybutler7](https://www.ruby-forum.com/u/johnnybutler7)\ **Post date:** [April 1, 2007, 6:14pm UTC](https://www.ruby-forum.com/t/indexing-mostly-binary-documents-ppt/90763/3 "2007-04-01T18:14:03Z")
On Apr 1, 2007, at 5:37 AM, Andreas K. wrote: > Are you serious? You’re adding raw, unprocessed PPT files to your > index? > > Now this is just wrong. PPT files may contain all sorts of binary > data, such as images and videos. I just had a look at the sample > presentation that came with my Office installation. This file is > 3.5MB in size with a (plain text) payload of less than 1KB. As I stated in my previous email, I am conjecturing that indexing these documents will not affect search performance. Do you disagree? > I’m sure there’s some tool available which converts PPT to plain text > and I strongly recommend you go out and find it. I’ve searched far and wide and have found none. john ---
**Author:** ![johnnybutler7](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/8491ac/32.png) [@johnnybutler7](https://www.ruby-forum.com/u/johnnybutler7)\ **Post date:** [April 1, 2007, 7:10pm UTC](https://www.ruby-forum.com/t/indexing-mostly-binary-documents-ppt/90763/4 "2007-04-01T19:10:10Z")
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, there are some: Catdoc and Antiword for example. Simple shell commands to extract text from files. [http://www.45.free.net/~vitus/software/catdoc/](http://www.45.free.net/~vitus/software/catdoc/) [http://www.winfield.demon.nl/](http://www.winfield.demon.nl/) Antiword has better windows support, but as far as I know doesn’t support .ppt as well as catdoc. I’m no expert though, just used it once or twice at the university. If you use them, I would be interested in feedback on how well it works. Thanks in advance and good luck Florian P.S.: There is an article about even more of them at [http://www.linux.com/article.pl?sid=06/02/22/201247](http://www.linux.com/article.pl?sid=06/02/22/201247) . John Joseph B. wrote: > these documents will not affect search performance. Do you disagree? > > * * * > > Ferret-talk mailing list > removed\_email\_address@domain.invalid > [http://rubyforge.org/mailman/listinfo/ferret-talk](http://rubyforge.org/mailman/listinfo/ferret-talk) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) Comment: Using GnuPG with Mozilla - [http://enigmail.mozdev.org](http://enigmail.mozdev.org) iD8DBQFGD+ID8RlGMqQ8m7oRAoFjAKCfgIzDsFnl+gKgnHQKI11yAkhTYQCfQpx3 fa5wJ2SaE2JlLzQABqxJe7Q= =AX5Q -----END PGP SIGNATURE----- ---
**Author:** ![johnnybutler7](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/8491ac/32.png) [@johnnybutler7](https://www.ruby-forum.com/u/johnnybutler7)\ **Post date:** [April 1, 2007, 8:04pm UTC](https://www.ruby-forum.com/t/indexing-mostly-binary-documents-ppt/90763/5 "2007-04-01T20:04:15Z")
> John Joseph B. wrote: > > > On Apr 1, 2007, at 5:37 AM, Andreas K. wrote: > > > > > I’m sure there’s some tool available which converts PPT to plain > > > text > > > and I strongly recommend you go out and find it. > > > > I’ve searched far and wide and have found none. On Apr 1, 2007, at 12:47 PM, Florian G. wrote: > once > or twice at the university. If you use them, I would be interested in > feedback on how well it works. Wow, I had never come across catdoc and its siblings, and believe me, I’ve searched far and wide. THANK YOU. btw-- I’m pretty sure antiword does not work on powerpoint: $ antiword powerpoint.ppt This OLE file does not contain a Word document John ---
**Author:** ![johnnybutler7](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/8491ac/32.png) [@johnnybutler7](https://www.ruby-forum.com/u/johnnybutler7)\ **Post date:** [April 2, 2007, 10:47am UTC](https://www.ruby-forum.com/t/indexing-mostly-binary-documents-ppt/90763/6 "2007-04-02T10:47:41Z")
On Apr 1, 2007, at 6:11 PM, John Joseph B. wrote: > > Now this is just wrong. PPT files may contain all sorts of binary > > data, such as images and videos. I just had a look at the sample > > presentation that came with my Office installation. This file is > > 3.5MB in size with a (plain text) payload of less than 1KB. > > As I stated in my previous email, I am conjecturing that indexing > these documents will not affect search performance. Do you disagree? I couldn’t disagree more. Question is to what extent does it affect performance. > > I’m sure there’s some tool available which converts PPT to plain text > > and I strongly recommend you go out and find it. > > I’ve searched far and wide and have found none. Seems like you found one now 🙂 Good Luck! – Andy ---
**Author:** ![johnnybutler7](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/8491ac/32.png) [@johnnybutler7](https://www.ruby-forum.com/u/johnnybutler7)\ **Post date:** [April 2, 2007, 1:30am UTC](https://www.ruby-forum.com/t/indexing-mostly-binary-documents-ppt/90763/7 "2007-04-02T01:30:45Z")
On Apr 1, 2007, at 12:47 PM, Florian G. wrote: > Catdoc and Antiword for example. Simple shell commands to extract text > from files. > > [http://www.45.free.net/~vitus/software/catdoc/](http://www.45.free.net/~vitus/software/catdoc/) > [http://www.winfield.demon.nl/](http://www.winfield.demon.nl/) > If you use them, I would be interested in > feedback on how well it works. I am now using catdoc, catppt, and xls2csv to index all of my documents, and it is working well. The content out of catppt seems to be rather incomplete, but is Good Enough for our purposes. John ---
**Author:** ![johnnybutler7](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/8491ac/32.png) [@johnnybutler7](https://www.ruby-forum.com/u/johnnybutler7)\ **Post date:** [April 2, 2007, 1:26pm UTC](https://www.ruby-forum.com/t/indexing-mostly-binary-documents-ppt/90763/8 "2007-04-02T13:26:01Z")
John B. wrote: > I am now using catdoc, catppt, and xls2csv to index all of my > documents, and it is working well. > > The content out of catppt seems to be rather incomplete, but is Good > Enough for our purposes. If you were going to be happy with the plain contents being indexed, I’d suggest just running the powerpoint document through strings before indexing it. I don’t know if catppt does more or less than that, but it’d be useful to compare. ---
**Author:** ![johnnybutler7](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/8491ac/32.png) [@johnnybutler7](https://www.ruby-forum.com/u/johnnybutler7)\ **Post date:** [April 2, 2007, 8:52pm UTC](https://www.ruby-forum.com/t/indexing-mostly-binary-documents-ppt/90763/9 "2007-04-02T20:52:11Z")
On Apr 2, 2007, at 7:02 AM, Alex Y. wrote: > If you were going to be happy with the plain contents being > indexed, I’d > suggest just running the powerpoint document through strings before > indexing it. I don’t know if catppt does more or less than that, but > it’d be useful to compare. I tried, that, the result is a LOT of binary garbage along with the plain text. ---
**Author:** ![johnnybutler7](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/8491ac/32.png) [@johnnybutler7](https://www.ruby-forum.com/u/johnnybutler7)\ **Post date:** [April 6, 2007, 10:21am UTC](https://www.ruby-forum.com/t/indexing-mostly-binary-documents-ppt/90763/10 "2007-04-06T10:21:49Z")
On 4/2/07, Andreas K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > performance. > Andy is right. Indexing binary data like this can really blow out the > size of an index. Indexing natural language you get a lot of common > terms so even in an index with millions of documents, you may have > only tens of thousands of terms. This has a natural compression effect > on the index so it will be a lot smaller than the collection of data > that is being indexed. This doesn’t work with binary data so the size > of your index will be much larger and you’ll have far more search > terms in the index. So it will definitely have an effect on search > performance but perhaps not as much as you’d expect. Nevertheless, > you’d be much better off extracting the text as others have already > said. Cheers, Dave ---
**Author:** ![johnnybutler7](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/j/8491ac/32.png) [@johnnybutler7](https://www.ruby-forum.com/u/johnnybutler7)\ **Post date:** [April 9, 2007, 6:12am UTC](https://www.ruby-forum.com/t/indexing-mostly-binary-documents-ppt/90763/11 "2007-04-09T06:12:53Z")
On Apr 6, 2007, at 4:02 AM, David B. wrote: > > performance. > > Andy is right. Indexing binary data like this can really blow out the > > size of an index. Indexing natural language you get a lot of common > > terms so even in an index with millions of documents, you may have > > only tens of thousands of terms. This has a natural compression effect > > on the index so it will be a lot smaller than the collection of data > > that is being indexed. This doesn’t work with binary data so the size > > of your index will be much larger and you’ll have far more search > > terms in the index. So it will definitely have an effect on search > > performance but perhaps not as much as you’d expect. For the record, by performance I meant the quality of the search (i.e., the results of a search query), and not the speed. I now realize that there is now way for anyone to have known that 🙂 Thanks again for all the ideas, I’m happy as a clam with catdoc/catppt. John