# Optimization help - reading out of /proc on Solaris **URL:** \ **Category:** Ruby\ **Created:** [September 16, 2008, 3:30pm UTC](https://www.ruby-forum.com/t/optimization-help-reading-out-of-proc-on-solaris/148717 "2008-09-16T15:30:58Z")\ **Posts on this page:** 5\ **Page:** 1

**Author:** ![danielb](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/c57346/32.png) [@danielb](https://www.ruby-forum.com/u/danielb)\ **Post date:** [September 16, 2008, 3:30pm UTC](https://www.ruby-forum.com/t/optimization-help-reading-out-of-proc-on-solaris/148717/1 "2008-09-16T15:30:58Z")

Hi all, Ruby 1.8.6 Solaris 10 I recently converted a C extension to get process table information on Solaris into a pure Ruby. I knew it would be slower, I just didn’t realize how _much_ slower it would be. I was expecting the pure Ruby version to be about 1/10th as fast. Instead, it’s about 1/70th as fast. Anticipating the, “Is it fast enough?” question, my answer is, “I’m not sure”. Besides, tuning can be fun. 🙂 Anyway, below is the code. I ran it through the profiler, but the top two most costly ops were Dir.foreach, which I don’t see any way to optimize\*, and the loop that gathers environment information, which I again see no way to optimize. # sunos.rb # # A pure Ruby version of sys-proctable for SunOS 5.8 or later #– # Directories under /proc on Solaris 2.8+ # The Sys module serves as a namespace only. module Sys # The ProcTable class encapsulates process table information. class ProcTable ``` # The version of the sys-proctable library VERSION = '0.8.0' private PRNODEV = -1 # non-existent device FIELDS = [ :flag, # process flags (deprecated) :nlwp, # number of active lwp's in the process :pid, # unique process id :ppid, # process id of parent :pgid, # pid of session leader :sid, # session id :uid, # real user id :euid, # effective user id :gid, # real group id :egid, # effective group id :addr, # address of the process :size, # size of process in kbytes :rssize, # resident set size in kbytes :ttydev, # tty device (or PRNODEV) :pctcpu, # % of recent cpu used by all lwp's :pctmem, # % of system memory used by process :start, # absolute process start time :time, # usr + sys cpu time for this process :ctime, # usr + sys cpu time for reaped children :fname, # name of the exec'd file :psargs, # initial characters argument list :wstat, # if a zombie, the wait status :argc, # initial argument count :argv, # address of initial argument vector :envp, # address of initial environment vector :dmodel, # data model of the process :taskid, # task id :projid, # project id :nzomb, # number of zombie lwp's in the process :poolid, # pool id :zoneid, # zone id :contract, # process contract :lwpid, # lwp id :wchan, # wait address for sleeping lwp :stype, # synchronization event type :state, # numeric lwp state :sname, # printable character for state :nice, # nice for cpu usage :syscall, # system call number (if in syscall) :pri, # priority :clname, # scheduling class name :name, # name of system lwp :onpro, # processor which last ran thsi lwp :bindpro, # processor to which lwp is bound :bindpset, # processor set to which lwp is bound :count, # number of contributing lwp's :tstamp, # current time stamp :create, # process/lwp creation time stamp :term, # process/lwp termination time stamp :rtime, # total lwp real (elapsed) time :utime, # user level cpu time :stime, # system call cpu time :ttime, # other system trap cpu time :tftime, # text page fault sleep time :dftime, # text page fault sleep time :kftime, # kernel page fault sleep time :ltime, # user lock wait sleep time :slptime, # all other sleep time :wtime, # wait-cpu (latency) time :stoptime, # stopped time :minf, # minor page faults :majf, # major page faults :nswap, # swaps :inblk, # input blocks :oublk, # output blocks :msnd, # messages sent :mrcv, # messages received :sigs, # signals received :vctx, # voluntary context switches :ictx, # involuntary context switches :sysc, # system calls :ioch, # chars read and written :path, # array of symbolic link paths from /proc// ``` pid :contracts, # array symbolic link paths from /proc// contracts :fd, # array of used file descriptors :cmd\_args, # array of command line arguments :environ # hash of environment associated with the process ] ``` public ProcTableStruct = Struct.new("ProcTableStruct", *FIELDS) # In block form, yields a ProcTableStruct for each process entry ``` that you # have rights to. This method returns an array of ProcTableStruct’s in # non-block form. # # If a +pid+ is provided, then only a single ProcTableStruct is yielded or # returned, or nil if no process information is found for that +pid+. # # Example: # # # Iterate over all processes # ProcTable.ps do |proc\_info| # p proc\_info # end # # # Print process table information for only pid 1001 # p ProcTable.ps(1001) # def self.ps(pid = nil) array = block\_given? ? nil : [] ``` Dir.foreach("/proc") do |file| next if file =~ /\D/ # Skip non-numeric entries under / ``` proc ``` # Only return information for a given pid, if provided if pid next unless file.to_i == pid end # Skip over any entries we don't have permissions to read begin psinfo = IO.read("/proc/#{file}/psinfo") rescue StandardError, Errno::EACCES next end struct = ProcTableStruct.new struct.flag = psinfo[0,4].unpack("i")[0] # pr_flag struct.nlwp = psinfo[4,4].unpack("i")[0] # pr_nlwp struct.pid = psinfo[8,4].unpack("i")[0] # pr_pid struct.ppid = psinfo[12,4].unpack("i")[0] # pr_ppid struct.pgid = psinfo[16,4].unpack("i")[0] # pr_pgid struct.sid = psinfo[20,4].unpack("i")[0] # pr_sid struct.uid = psinfo[24,4].unpack("i")[0] # pr_uid struct.euid = psinfo[28,4].unpack("i")[0] # pr_euid struct.gid = psinfo[32,4].unpack("i")[0] # pr_gid struct.egid = psinfo[36,4].unpack("i")[0] # pr_egid struct.addr = psinfo[40,4].unpack("L")[0] # pr_addr struct.size = psinfo[44,4].unpack("L")[0] * 1024 # ``` pr\_size struct.rssize = psinfo[48,4].unpack(“L”)[0] \* 1024 # pr\_rssize ``` # skip pr_pad1 # TODO: Convert this to a human readable string somehow struct.ttydev = psinfo[56,4].unpack("i")[0] # pr_ttydev # pr_pctcpu struct.pctcpu = (psinfo[60,2].unpack("S")[0] * 100).to_f / ``` 0x8000 ``` # pr_pctmem struct.pctmem = (psinfo[62,2].unpack("S")[0] * 100).to_f / ``` 0x8000 ``` struct.start = Time.at(psinfo[64,8].unpack("L")[0]) # ``` pr\_start struct.time = psinfo[72,8].unpack(“L”)[0] # pr\_time struct.ctime = psinfo[80,8].unpack(“L”)[0] # pr\_ctime ``` struct.fname = psinfo[88,16].strip # pr_fname struct.psargs = psinfo[104,80].strip # pr_psargs struct.wstat = psinfo[184,4].unpack("i")[0] # pr_wstat struct.argc = psinfo[188,4].unpack("i")[0] # pr_argc struct.argv = psinfo[192,4].unpack("L")[0] # pr_argv struct.envp = psinfo[196,4].unpack("L")[0] # pr_envp struct.dmodel = psinfo[200,1].unpack("C")[0] # pr_dmodel # skip pr_pad2 struct.taskid = psinfo[204,4].unpack("i")[0] # pr_taskid struct.projid = psinfo[208,4].unpack("i")[0] # ``` pr\_projectid struct.nzomb = psinfo[212,4].unpack(“i”)[0] # pr\_nzomb struct.poolid = psinfo[216,4].unpack(“i”)[0] # pr\_poolid struct.zoneid = psinfo[220,4].unpack(“i”)[0] # pr\_zoneid struct.contract = psinfo[224,4].unpack(“i”)[0] # pr\_contract ``` # skip pr_filler ### lwpsinfo struct info # skip pr_flag struct.lwpid = psinfo[236,4].unpack("i")[0] # pr_lwpid # skip pr_addr struct.wchan = psinfo[244,4].unpack("L")[0] # pr_wchan struct.stype = psinfo[248,1].unpack("C")[0] # pr_stype struct.state = psinfo[249,1].unpack("C")[0] # pr_state struct.sname = psinfo[250,1] # pr_sname struct.nice = psinfo[251,1].unpack("C")[0] # pr_nice struct.syscall = psinfo[252,2].unpack("S")[0] # pr_syscall # skip pr_oldpri # skip pr_cpu struct.pri = psinfo[256,4].unpack("i")[0] # pr_syscall # skip pr_pctcpu # skip pr_pad # skip pr_start # skip pr_time struct.clname = psinfo[280,8].strip # pr_clname struct.name = psinfo[288,16].strip # pr_name struct.onpro = psinfo[304,4].unpack("i")[0] # pr_onpro struct.bindpro = psinfo[308,4].unpack("i")[0] # ``` pr\_bindpro struct.bindpset = psinfo[308,4].unpack(“i”)[0] # pr\_bindpset ``` # Get the full command line out of /proc//as. begin fd = File.open("/proc/#{file}/as") fd.sysseek(struct.argv, IO::SEEK_SET) address = fd.sysread(struct.argc * 4).unpack("L")[0] struct.cmd_args = [] 0.upto(struct.argc - 1){ |i| fd.sysseek(address, IO::SEEK_SET) data = fd.sysread(128)[/^[^\0]*/] # Null strip struct.cmd_args << data address += data.length + 1 # Add 1 for the space } # Get the environment hash associated with the process. struct.environ = {} fd.sysseek(struct.envp, IO::SEEK_SET) env_address = fd.sysread(128).unpack("L")[0] loop do fd.sysseek(env_address, IO::SEEK_SET) data = fd.sysread(1024)[/^[^\0]*/] # Null strip break if data.empty? key, value = data.split('=') struct.environ[key] = value env_address += data.length + 1 # Add 1 for the space end rescue Errno::EACCES, Errno::EOVERFLOW, EOFError # Skip this if we don't have proper permissions, if ``` there’s # no associated environment, or if there’s a largefile issue. ensure fd.close if fd end ``` ### struct prusage begin prusage = 0.chr * 512 prusage = IO.read("/proc/#{file}/usage") # skip pr_lwpid struct.count = prusage[4,4].unpack("i")[0] # ``` pr\_count struct.tstamp = prusage[8,8].unpack(“L”)[0] # pr\_tstamp struct.create = prusage[16,8].unpack(“L”)[0] # pr\_create struct.term = prusage[24,8].unpack(“L”)[0] # pr\_term struct.rtime = prusage[32,8].unpack(“L”)[0] # pr\_rtime struct.utime = prusage[40,8].unpack(“L”)[0] # pr\_utime struct.stime = prusage[48,8].unpack(“L”)[0] # pr\_stime struct.ttime = prusage[56,8].unpack(“L”)[0] # pr\_ttime struct.tftime = prusage[64,8].unpack(“L”)[0] # pr\_tftime struct.dftime = prusage[72,8].unpack(“L”)[0] # pr\_dftime struct.kftime = prusage[80,8].unpack(“L”)[0] # pr\_kftime struct.ltime = prusage[88,8].unpack(“L”)[0] # pr\_ltime struct.slptime = prusage[96,8].unpack(“L”)[0] # pr\_slptime struct.wtime = prusage[104,8].unpack(“L”)[0] # pr\_wtime struct.stoptime = prusage[112,8].unpack(“L”)[0] # pr\_stoptime struct.minf = prusage[120,4].unpack(“L”)[0] # pr\_minf struct.majf = prusage[124,4].unpack(“L”)[0] # pr\_majf struct.nswap = prusage[128,4].unpack(“L”)[0] # pr\_nswap struct.inblk = prusage[128,4].unpack(“L”)[0] # pr\_inblk struct.oublk = prusage[128,4].unpack(“L”)[0] # pr\_oublk struct.msnd = prusage[128,4].unpack(“L”)[0] # pr\_msnd struct.mrcv = prusage[128,4].unpack(“L”)[0] # pr\_mrcv struct.sigs = prusage[128,4].unpack(“L”)[0] # pr\_sigs struct.vctx = prusage[128,4].unpack(“L”)[0] # pr\_vctx struct.ictx = prusage[128,4].unpack(“L”)[0] # pr\_ictx struct.sysc = prusage[128,4].unpack(“L”)[0] # pr\_sysc struct.ioch = prusage[128,4].unpack(“L”)[0] # pr\_ioch rescue Errno::EACCES # Do nothing if we lack permissions. Just move on. end ``` # Information from /proc//path. This is represented ``` as a hash, # with the symbolic link name as the key, and the file it links to # as the value, or nil if it cannot be found. #– # Note that cwd information can be gathered from here, too. struct.path = {} ``` Dir["/proc/#{file}/path/*"].each{ |entry| link = File.readlink(entry) rescue nil struct.path[File.basename(entry)] = link } # Information from /proc//contracts. This is ``` represented as # a hash, with the symbolic link name as the key, and the file # it links to as the value. struct.contracts = {} ``` Dir["/proc/#{file}/contracts/*"].each{ |entry| link = File.readlink(entry) rescue nil struct.contracts[File.basename(entry)] = link } # Information from /proc//fd. This returns an array ``` of # numeric file descriptors used by the process. struct.fd = Dir["/proc/#{file}/fd/\*"].map{ |f| File.basename(f).to\_i } ``` if block_given? yield struct else array << struct end end pid ? array[0] : array end ``` end end Thanks, Dan - I tried tossing threads at it in a one-thread-per-directory approach, but they didn’t get along with IO.read in MRI, and seemed to provide no real speed benefit with JRuby. ---

**Author:** ![danielb](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/c57346/32.png) [@danielb](https://www.ruby-forum.com/u/danielb)\ **Post date:** [September 16, 2008, 4:49pm UTC](https://www.ruby-forum.com/t/optimization-help-reading-out-of-proc-on-solaris/148717/2 "2008-09-16T16:49:34Z")

> Anyway, below is the code. I ran it through the profiler, but the top > two most costly ops were Dir.foreach, which I don’t see any way to > optimize\*, and the loop that gathers environment information, which I > again see no way to optimize. Could you post your profiling? If you run using “time”, how much user CPU versus system CPU are you using? Have you tried using Dir.open.each instead of Dir["/foo/\*"]? Maybe globbing is expensive. Your environment loop does a fresh sysread(1024) for each var=val pair, even if you’ve only consumed (say) 7 bytes from the previous call. You would make many fewer system calls if you read a big chunk and chopped it up afterwards. This may also avoid non-byte-aligned reads. I would also be tempted to write one long unpack instead of lots of string slicing and unpacking. The overhead here may be negligible, but the code may end up being smaller and simpler. e.g. struct = ProcTableStruct.new(\*psinfo.unpack(\<\ **Author:** ![danielb](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/c57346/32.png) [@danielb](https://www.ruby-forum.com/u/danielb)\ **Post date:** [September 16, 2008, 5:25pm UTC](https://www.ruby-forum.com/t/optimization-help-reading-out-of-proc-on-solaris/148717/3 "2008-09-16T17:25:58Z") 2008/9/16 Daniel B. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > again see no way to optimize. > You can optimize the loop body. Profiler output takes a while to get > used to. Few things that caught my attention: You can combine the first two “next” in one. Also matching with the RX on the left side is faster AFAIK. Try to use as few psinfo.unpack as possible, i.e. ideally only 1. Use the block form of File.open. Do not use sysread/syswrite/sysseek unless you have to (most of the time you don’t). You can replace all but the first checks for block\_given? with “array”. Might be faster. Have fun! robert ---

**Author:** ![danielb](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/c57346/32.png) [@danielb](https://www.ruby-forum.com/u/danielb)\ **Post date:** [September 19, 2008, 12:55am UTC](https://www.ruby-forum.com/t/optimization-help-reading-out-of-proc-on-solaris/148717/4 "2008-09-19T00:55:31Z")

Brian C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Your environment loop does a fresh sysread(1024) for each var=val pair, > even if you’ve only consumed (say) 7 bytes from the previous call. You > would make many fewer system calls if you read a big chunk and chopped > it up afterwards. This may also avoid non-byte-aligned reads. > > I would also be tempted to write one long unpack instead of lots of > string slicing and unpacking. The overhead here may be negligible, but > the code may end up being smaller and simpler. e.g. Method calls are expensive. Right now he’s got 3 per field (array indexing, unpack, and assignment). He could get that down to 1 per field (assignment only) pretty easily by following your suggestion. ---

**Author:** ![danielb](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/c57346/32.png) [@danielb](https://www.ruby-forum.com/u/danielb)\ **Post date:** [September 19, 2008, 12:56am UTC](https://www.ruby-forum.com/t/optimization-help-reading-out-of-proc-on-solaris/148717/5 "2008-09-19T00:56:07Z")

Brian C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > i i i i > [:nlwp,“i”], # number of active lwp’s in the process > … > [:size,“s”], # size of process in kbytes > [:rssize,“s”], # resident set size in kbytes > [nil,“X4”], # skip pr\_pad1 > … etc > > HTH, > > Brian. Here’s a concept for metaprogramming that that I was able to generate mostly by running regexes to transform the code. I’ve only tackled /proc/#{file}/psinfo, but it should be fairly simple to extend to the other files as well # The Sys module serves as a namespace only. module Sys # The ProcTable class encapsulates process table information. class ProcTable ``` # The version of the sys-proctable library VERSION = '0.8.0' private PRNODEV = -1 # non-existent device #Dissecting the format of this, we have a symbol mapping to an ``` unpack format string segment #the @ sign followed by a number indicates the offset in the string, and the text following that number is the format #of the data to unpack FIELDS=[ [:flag , “@0 i”], [:nlwp , “@4 i”], [:pid , “@8 i”], [:ppid , “@12 i”], [:pgid , “@16 i”], [:sid , “@20 i”], [:uid , “@24 i”], [:euid , “@28 i”], [:gid , “@32 i”], [:egid , “@36 i”], [:addr , “@40 L”], [:size , “@44 L”], [:rssize , “@48 L”], [:ttydev , “@56 i”], [:pctcpu , “@60 S”], [:pctmem , “@62 S”], [:start , “@64 L”], [:time , “@72 L”], [:ctime , “@80 L”], #note that the A format specifier automatically does what the #strip method does #so I don’t have to call .strip in the ps method [:fname , “@88 A16”], [:psargs , “@104 A80”], [:wstat , “@184 i”], [:argc , “@188 i”], [:argv , “@192 L”], [:envp , “@196 L”], [:dmodel , “@200 C”], [:taskid , “@204 i”], [:projid , “@208 i”], [:nzomb , “@212 i”], [:poolid , “@216 i”], [:zoneid , “@220 i”], [:contract , “@224 i”], [:lwpid , “@236 i”], [:wchan , “@244 L”], [:stype , “@248 C”], [:state , “@249 C”], [:sname , “@250 a1”], [:nice , “@251 C”], [:syscall , “@252 S”], [:pri , “@256 i”], [:clname , “@280 A8”], [:name , “@288 A16”], [:onpro , “@304 i”], [:bindpro , “@308 i”], [:bindpset , “@308 i”] ] ``` field_names,format_strings=FIELDS.transpose eval <<-"end;" def first_pass_fill string struct=ProcTableStruct.new #{ field_names.collect{|x| "struct.#{x}"}.join(", ") } = ``` string.unpack “#{format\_strings.join ’ '}” end end; ``` #repeat the above with a new array instead of FIELDS and a new ``` method name #for any other file you want to unpack this way =begin This eval will define a function with the following code. The arrays and metaprogramming are just an easier way to manage the format string and fieldnames that you can understand them when maintenence time comes around. ``` def first_pass_fill string struct=ProcTableStruct.new struct.flag, struct.nlwp, struct.pid, struct.ppid, ``` struct.pgid, struct.sid, struct.uid, struct.euid, struct.gid, struct.egid, struct.addr, struct.size, struct.rssize, struct.ttydev, struct.pctcpu, struct.pctmem, struct.start, struct.time, struct.ctime, struct.fname, struct.psargs, struct.wstat, struct.argc, struct.argv, struct.envp, struct.dmodel, struct.taskid, struct.projid, struct.nzomb, struct.poolid, struct.zoneid, struct.contract, struct.lwpid, struct.wchan, struct.stype, struct.state, struct.sname, struct.nice, struct.syscall, struct.pri, struct.clname, struct.name, struct.onpro, struct.bindpro, struct.bindpset = string.unpack “@0 i @4 i @8 i @12 i @16 i @20 i @24 i @28 i @32 i @36 i @40 L @44 L @48 L @56 i @60 S @62 S @64 L @72 L @80 L @88 A16 @104 A80 @184 i @188 i @192 L @196 L @200 C @204 i @208 i @212 i @216 i @220 i @224 i @236 i @244 L @248 C @249 C @250 a1 @251 C @252 S @256 i @280 A8 @288 A16 @304 i @308 i @308 i” end =end ``` public ProcTableStruct = Struct.new("ProcTableStruct", *field_names) #if you have multiple files you're reading from with their field ``` names #in multiple different variables, you’ll want to replace field\_names #with some array concatentation ``` # In block form, yields a ProcTableStruct for each process entry ``` that you # have rights to. This method returns an array of ProcTableStruct’s in # non-block form. # # If a +pid+ is provided, then only a single ProcTableStruct is yielded or # returned, or nil if no process information is found for that +pid+. # # Example: # # # Iterate over all processes # ProcTable.ps do |proc\_info| # p proc\_info # end # # # Print process table information for only pid 1001 # p ProcTable.ps(1001) # def self.ps(pid = nil) array = block\_given? ? nil : [] Dir.foreach(“/proc”) do |file| next if file =~ \D # Skip non-numeric entries under / proc ``` # Only return information for a given pid, if provided if pid next unless file.to_i == pid end # Skip over any entries we don't have permissions to read begin psinfo = IO.read("/proc/#{file}/psinfo") rescue StandardError, Errno::EACCES next end #the first pass fill just gets the raw data and unpacks it struct = first_pass_fill psinfo #now we do the transformations we need on the few fields ``` that need it struct.pctcpu= (struct.pctcpu_100).to\_f / 0x8000 struct.pctmem= (struct.pctmem_100).to\_f / 0x8000 struct.start=Time.at(struct.start) #the fields that needed stripping were handled by unpack ``` #repeat the above for other files that we need to deal with if block_given? yield struct else array << struct end end pid ? array[0] : array end ``` end end