Pretty folder tree script

john_old · April 24, 2008, 11:07pm

Hola,
My boss asked me to make a nice map of the directories on our web
server for an upcoming meeting. I’m aware of several ways to approach
this, like find and tree, etc, but I’m a Ruby addict, so that’s what
I’m using. This gets good, don’t bail yet!

What I want to see looks like this:

Foldername

file1 file2

Nested Foldername

file3 file4

Thus, my nested folders appear nested on the page.

I start by doing a “ls -R > filemap.txt” on the directory I’m
interested in, and then I can process the output file:

map_file.rb

final_document = File.open(‘newmap.html’, ‘w’) do |f|
f << “\n”
f << “<!-- \n.folder { \n”
f << " background-color: #99CCCC; \nborder: 1px solid #333333;
\ndisplay: block; \nmargin-top: 3px; \nmargin-right: 3px; \nmargin-
left: 6px; \nmargin-bottom: 6px; \n}\n "
f << “.title {\n”
f << " background-color: #D5EAEA;\n width: 99%;\n padding-left: 1%;
\nborder-bottom-width: 1px; \nborder-bottom-style: dotted;\n border-
bottom-color: #333333; \n}\n"
f << “.file {\n”
f << “padding-right: 12px; \n padding-left: 12px;”
f << “\n}\n”
f << “”
f << “Version 19:
\n”
f << “

\n\t”
File.open(‘filemap.txt’).each do |x|
case x
when /.*:$/
f << “\n

\n\t”
f << x
f << “\n\t

”
when /^\n/
f << “

”
else
f << “\n\n\t”
f << x
f << “\n\t”
end
end
f << “

”

f << “\n”
end

If you run this, everything goes fine – except for the nesting part.
I don’t think I can solve this one in a case statement - I think I’m
going to need an array like:

arrayname[folder, folder[child]]

and iterate over that, putting my at the end of each nested
array.

I’m just not that good! This one is killing me. Can anyone help?

Mahalo,
John

john_old · April 25, 2008, 2:26am

Well, I’ve gotten this far:

[[".", “Migratus:”], [".", “Migratus”, “app:”], [".", “Migratus”,
“app”, “controllers:”], [".", “Migratus”, “app”, “helpers:”], [".",
“Migratus”, “app”, “models:”], [".", “Migratus”, “app”, “views:”],
[".", “Migratus”, “app”, “views”, “layouts:”], [".", “Migratus”,
“app”, “views”, “projects:”], [".", “Migratus”, “app”, “views”,
“tasks:”], [".", “Migratus”, “backup-sunday:”], [".", “Migratus”,
“config:”]]

Maybe someone sees how I can transform this into the nested array I
need. If so, your help is appriciated.

john_old · April 25, 2008, 2:48am

Hi,

On Fri, Apr 25, 2008 at 10:25 AM, John [email protected] wrote:

Maybe someone sees how I can transform this into the nested array I
need. If so, your help is appriciated.

celtic@sohma:~$ cat migratus.rb
def transform_nested data
result = {:name => “root”, :elements => []}
data.each do |item|
index = result
item.each do |sub|
sub.gsub! /:$/, ‘’
new_index = index[:elements].find {|e| e[:name] == sub}
unless new_index
new_index = {:name => sub, :elements => []}
index[:elements] << new_index
end
index = new_index
end
end
result
end

data = [[“.”, “Migratus:”], [“.”, “Migratus”, “app:”], [“.”, “Migratus”,
“app”, “controllers:”], [“.”, “Migratus”, “app”, “helpers:”], [“.”,
“Migratus”, “app”, “models:”], [“.”, “Migratus”, “app”, “views:”],
[“.”, “Migratus”, “app”, “views”, “layouts:”], [“.”, “Migratus”,
“app”, “views”, “projects:”], [“.”, “Migratus”, “app”, “views”,
“tasks:”], [“.”, “Migratus”, “backup-sunday:”], [“.”, “Migratus”,
“config:”]]

require ‘pp’
pp transform_nested(data)
celtic@sohma:~$ ruby migratus.rb
{:elements=>
[{:elements=>
[{:elements=>
[{:elements=>
[{:elements=>[], :name=>“controllers”},
{:elements=>[], :name=>“helpers”},
{:elements=>[], :name=>“models”},
{:elements=>
[{:elements=>[], :name=>“layouts”},
{:elements=>[], :name=>“projects”},
{:elements=>[], :name=>“tasks”}],
:name=>“views”}],
:name=>“app”},
{:elements=>[], :name=>“backup-sunday”},
{:elements=>[], :name=>“config”}],
:name=>“Migratus”}],
:name=>“.”}],
:name=>“root”}
celtic@sohma:~$

The data may be hard to visualise in that way (I also stripped the
ending
:'s from some items so there were no duplicates), but basically we have the result item result’:

result[:name] == “root” [the root object]
result[:elements] contains the item with [:name] == “.”, which in turn
has
[:elements] with the item with name “Migratus”, which has “app”,
“backup-sunday”, “config” - so, nested hashes and arrays, which you can
traverse to your own delight.

HTH!

Arlen

john_old · April 25, 2008, 2:50am

Hi,

Apologies; I should have used the obvious solution and pastebinned it:
http://pastie.caboo.se/186519

Cheers,
Arlen.

john_old · April 25, 2008, 3:36am

I think you killed a fly with a shotgun this time. I mean it’s really
appriciated, but I can’t ride this horse.

I was looking for more of a [migratus[app[views, models,
controllers]]… etc] than a hash with empty symbols.

john_old · April 25, 2008, 3:48am

Hi,

On Fri, Apr 25, 2008 at 11:35 AM, John [email protected] wrote:

I think you killed a fly with a shotgun this time. I mean it’s really
appriciated, but I can’t ride this horse.

Understood, but I think this is probably the simplest data structure you
can
get for what you want.

I was looking for more of a [migratus[app[views, models,
controllers]]… etc] than a hash with empty symbols.

You can’t use anything less than arrays and something else to do this,
unfortunately - since arrays can only contain other objects [and have no
name' of their own], we need some other type of object used in conjunction to actually be the nodes’ of this tree structure. You could do it
entirely
with hashes, however, which renders what I think what you want:

result[:migratus][:app][:views], for example. Is this more what you’d be
looking for? My last question, though, is - what do you want at the
leaves' of this tree? [i.e. what do you get at the very end? - the items under views’, for example?] Something that produces such nested hashes would
just
leave empty hashes (i.e. result[…][:views][:abc] == {}), but that
wouldn’t
impede your work.

The only thing with using hashes is that you lose any order you might’ve
had. But perhaps it’s not a problem?

http://pastie.caboo.se/186549

The result is like this:

{“controllers”=>{},
“views”=>{“projects”=>{}, “tasks”=>{}, “layouts”=>{}},
“helpers”=>{},
“models”=>{}}

Note that we’re already “using” the data in some manipulated sense in
that
pastie bit by asking for `result[“.”][“Migratus”][“app”]’ specifically.

Hope this helps a bit. If I miss the point still, please let us know!

Arlen

john_old · April 25, 2008, 4:05am

hm. Well, here’s my code so far:

http://pastie.caboo.se/186554

If you run that you’ll see how I’m tripped up.

john_old · April 25, 2008, 4:53am

Hi,

On Fri, Apr 25, 2008 at 12:50 PM, John [email protected] wrote:

I must be clueless - this doesn’t work either:

@map.each do |i|
i.each do |k, v|
f << “#{k} is: #{v}
”
end
end

The way the code is currently set out won’t really work at all with it -
I’ve been working on a replacement for that little loop of yours - I’ll
post
it as soon as I’m done!

Cheers,
Arlen

john_old · April 25, 2008, 4:51am

I must be clueless - this doesn’t work either:

@map.each do |i|
i.each do |k, v|
f << “#{k} is: #{v}
”
end
end

john_old · April 25, 2008, 5:01am

Hi John,

http://pastie.caboo.se/186576

Replace the part after the test data with this. It produces the output
I’d
expect to see! I added some brief comments so hopefully you can get an
idea
of what it’s doing.

The next part would probably be sorting, at least at a rudimentary
level, so
that we process all the files first, then directories (at one level) -
at
the moment, directories and files are interspersed within the same
level, so
it’s hard to get an overall idea. But perhaps you can work that out!

Cheers,
Arlen

john_old · April 25, 2008, 11:15am

Hi,

On Fri, Apr 25, 2008 at 7:10 PM, Eivind E. [email protected]
wrote:

recurse_dir(“/wherever/you/start”)

Eivind.

I was too caught up in the code that I wrote a complex solution to a
simple
problem.

You attacked it from its roots! Excellent stuff.

Arlen

john_old · April 25, 2008, 9:05pm

Something like this should work:

Implement this properly

def html_escape(s)
s
end

Why are we escaping html? It’s just a list of file names, after all.

end
end
end

recurse_dir("/wherever/you/start")

Eivind.

Your code looks good, but it just loops…

trimmed… *

.

file_map_2.rb:7:in `recurse_dir': stack level too deep (SystemStackError) from file_map_2.rb:11:in `recurse_dir' from file_map_2.rb:7:in `each' from file_map_2.rb:7:in `recurse_dir' from file_map_2.rb:11:in `recurse_dir' from file_map_2.rb:7:in `each' from file_map_2.rb:7:in `recurse_dir' from file_map_2.rb:11:in `recurse_dir' from file_map_2.rb:7:in `each' ... 2311 levels... from file_map_2.rb:11:in `recurse_dir' from file_map_2.rb:7:in `each' from file_map_2.rb:7:in `recurse_dir' from file_map_2.rb:19

Looks like it should work to me. I don’t see how it can be infinite.

john_old · April 25, 2008, 11:11am

On Thu, Apr 24, 2008 at 11:05 PM, John [email protected] wrote:

<div class='folder'>
<div class='title'> Nested Foldername </div>
<span class='file'> file3</span><span class='file'> file4</span>
Thus, my nested folders appear nested on the page.

I start by doing a “ls -R > filemap.txt” on the directory I’m
interested in, and then I can process the output file:

I think this is your core problem. Don’t do that.

Something like this should work:

Implement this properly

def html_escape(s)
s
end

def recurse_dir(dirname)
Dir.entries(dirname).sort.each do |filename|
if FileTest.directory? filename
puts “<div class="folder">”
puts “<div class="title">#{html_escape(filename)}”
recurse_dir(filename)
puts “”
elsif FileTest.file? filename
puts “<span class="file">#{html_escape(filename)}”
end
end
end

recurse_dir(“/wherever/you/start”)

Eivind.

john_old · April 25, 2008, 9:46pm

So now it looks like (ascii rawx!!1)

.-------------------
| folder A title
| .------------------
| | folder B title
| ----------------- | child_file_of_A | .------------------- | | folder C | | child_file_of_C |-------------------
`-----------------

example.sort.each just smashes the whole thing to bits. What the F do
I do now?

john_old · April 25, 2008, 9:17pm

You attacked it from its roots! Excellent stuff.

Arlen

Arlen, your solution works. This is a complex problem that appears
simple.

john_old · April 26, 2008, 12:49am

Hi,

On Sat, Apr 26, 2008 at 5:45 AM, John [email protected] wrote:

| | child_file_of_C
| ------------------- -----------------

example.sort.each just smashes the whole thing to bits. What the F do
I do now?

I think Eivind’s solution definitely has some merit, perhaps it’s just a
simple bug that needs to be fixed. I’ll look at it tonight.

For this problem, you can’t sort example' - you need to work on the data after it's been collected into the hashes. I don't have the time to implement it right now, but basically, as we look at each directory, we convert the hash we'd do the blah.each’ on into an Array instead:

{:q => 2, :c => 4}.to_a
=> [[:c, 4], [:q, 2]]

Then we can sort this array in whatever way we like:

{:q => 2, :c => 4}.to_a.sort_by {|a| a[1]}
=> [[:q, 2], [:c, 4]]

Notice in this example I sorted by' a[1] for every element - that is, it used the 1th item of each element as the index. (in this case, the numbers 2 and 4) If I said a[0]‘, it would’ve tried to use the symbols (and you
can’t
sort by symbol). If we wanted them in the order :c', :q’, we could
order
by a[0].to_s, if we like:

{:q => 2, :c => 4}.to_a.sort_by {|a| a[0].to_s}
=> [[:c, 4], [:q, 2]]

This is just an example of how you could do it. Maybe you can work it
out
from here. If not, I’ll be back tonight and I’ll take a look at Eivind’s
method and how you can sort properly with that, too.

Cheers,
Arlen

john_old · April 26, 2008, 1:30am

Hi,

On Sat, Apr 26, 2008 at 9:05 AM, John [email protected] wrote:

Where Eivind is correct is the way the function calls itself. I think
changing

if FileTest.directory? filename

to something like

if x =~ /.*:$/

might work. Then again, the real problem still isn’t solved.

Nope. Eivind pointed out that your usage of the output of ls -R is
incorrect
in the first place, which is correct - that function evaluates the
directory
structure on its own, which is the correct (and much simpler) thing to
do.
It’s just that Eivind forgot to ignore the .' and …’ entries.

Here’s the fixed recurse_dir:

def recurse_dir(o, dirname)
entries = Dir.entries(dirname).sort
directories = entries.find_all {|e| FileTest.directory?
File.join(dirname,
e)}.sort
files = entries.find_all {|e| FileTest.file? File.join(dirname,
e)}.sort

directories.reject! {|a| a =~ /^./}
files.reject! {|a| a =~ /^./}

files.each do |filename|
o.puts “<span class="file">#{html_escape(filename)}”
end

directories.each do |filename|
o.puts “<div class="folder">”
o.puts “<div class="title">#{html_escape(filename)}”
recurse_dir o, File.join(dirname, filename)
o.puts “”
end
end

Note that it ignores filenames/directorynames starting with a .', which avoids the recursion problem. It also lists files first, then directories (both sorted). Use this instead of the entire bit where you process example’.

Here’s the entire script pieced back together:

http://pastie.caboo.se/187050

It’s a lot leaner (without that sample output, I guess). I just tested
this
on my home directory: I have rather a lot of files, it produced a 2.0MB
file
in about a minute, which is pretty impressive, I think. And it’s not so
unreadable, though maybe the styles could use some work.

So there we have it. Tell me how it works out for you. This was mostly
Eivind’s inspiration, though.

Arlen

john_old · April 26, 2008, 2:15am

I am especially thankful and appreciative of all the work that you
both have put into this script. I never would have written it this
well. Thank you Eivind and Arlen, I owe you one.

It works perfectly, of course.

I can’t run a ruby script on the server that I need to map, which is
why I did “ls -R > output.txt” in the first place. So: does it work or
doesn’t it? Yes.

john_old · April 26, 2008, 4:25am

I updated the CSS and made it more readable. I’ll work on this over
the weekend: it simple must work with a plain text file, even if ‘do
it in Ruby’ is cleaner and better. Also, it turns out that once you
recurse, oh, I dunno, 8 levels you have no idea where the file
actually lives. I have tp add a bit of formatting for that, too.

I’m really impressed, overall. (… not with MY code, with yours.)

Here’s the code:
http://pastie.caboo.se/187098

john_old · April 26, 2008, 1:06am

I think Eivind’s solution definitely has some merit, perhaps it’s just a
simple bug that needs to be fixed. I’ll look at it tonight.

Well, it really wont, because

if FileTest.directory? filename

is going to throw up on a plain text file. We are, after all,
processing the textfile output of ls -R

Where Eivind is correct is the way the function calls itself. I think
changing

if FileTest.directory? filename

to something like

if x =~ /.*:$/

might work. Then again, the real problem still isn’t solved.