Wrapping a C struct[] constant

I have some C code with a struct definition

struct flag_str {
unsigned int val;
const char *str;
};

and an inline array

struct flag_str extent_flags[] = {
{ FIEMAP_EXTENT_LAST, “last” },
{ FIEMAP_EXTENT_UNKNOWN, “unkown” },
{ FIEMAP_EXTENT_DELALLOC, “delalloc” },
{ FIEMAP_EXTENT_NO_BYPASS, “no_bypass” },
{ FIEMAP_EXTENT_SECONDARY, “secondary” },
{ FIEMAP_EXTENT_NET, “net” },
{ FIEMAP_EXTENT_DATA_COMPRESSED, “data_compressed” },
{ FIEMAP_EXTENT_DATA_ENCRYPTED, “data_encrypted” },
{ FIEMAP_EXTENT_NOT_ALIGNED, “not_aligned” },
{ FIEMAP_EXTENT_DATA_INLINE, “data_inline” },
{ FIEMAP_EXTENT_DATA_TAIL, “data_tail” },
{ FIEMAP_EXTENT_UNWRITTEN, “unwritten” },
{ FIEMAP_EXTENT_MERGED, “merged” },
{ 0, NULL },
};

What’s the simplest way to expose that array as a ruby constant (other
than just copy/pasting the data, of course :))?

martin

Martin DeMello wrote:

I have some C code with a struct definition

struct flag_str {
unsigned int val;
const char *str;
};

and an inline array

struct flag_str extent_flags[] = {
{ FIEMAP_EXTENT_LAST, “last” },
{ FIEMAP_EXTENT_UNKNOWN, “unkown” },
{ FIEMAP_EXTENT_DELALLOC, “delalloc” },
{ FIEMAP_EXTENT_NO_BYPASS, “no_bypass” },
{ FIEMAP_EXTENT_SECONDARY, “secondary” },
{ FIEMAP_EXTENT_NET, “net” },
{ FIEMAP_EXTENT_DATA_COMPRESSED, “data_compressed” },
{ FIEMAP_EXTENT_DATA_ENCRYPTED, “data_encrypted” },
{ FIEMAP_EXTENT_NOT_ALIGNED, “not_aligned” },
{ FIEMAP_EXTENT_DATA_INLINE, “data_inline” },
{ FIEMAP_EXTENT_DATA_TAIL, “data_tail” },
{ FIEMAP_EXTENT_UNWRITTEN, “unwritten” },
{ FIEMAP_EXTENT_MERGED, “merged” },
{ 0, NULL },
};

What’s the simplest way to expose that array as a ruby constant (other
than just copy/pasting the data, of course :))?

How/where are you using it?

FFI is generally a good solution, although it
would probably be best to actually define the
Array on the Ruby side and pass it in as needed.

Eero

Magic is insufficiently advanced technology.

On Fri, Jun 12, 2009 at 7:42 AM, Eero S.
[email protected]wrote:

struct flag_str extent_flags[] = {
{ FIEMAP_EXTENT_DATA_TAIL, “data_tail” },
FFI is generally a good solution, although it

Even using FFI, you’ll need to define the wrapping between C constant to
Ruby constant, and using Ruby’s API, you’ll be define_const-ing every
one of
those, then putting them into an array. Just define them in the Ruby
yourself, there isn’t a faster way to do it.

Jason

On Fri, Jun 12, 2009 at 5:12 PM, Eero S.[email protected]
wrote:

struct flag_str extent_flags[] = {
{ FIEMAP_EXTENT_DATA_TAIL, “data_tail” },
{ FIEMAP_EXTENT_UNWRITTEN, “unwritten” },
{ FIEMAP_EXTENT_MERGED, “merged” },
{ 0, NULL },
};

What’s the simplest way to expose that array as a ruby constant (other
than just copy/pasting the data, of course :))?

How/where are you using it?

Using it (or want to use it!) from the ruby side to generate a hash of
constant => name and use that to unpack and display flag settings.

m.

On Fri, Jun 12, 2009 at 7:01 PM, Jason R.[email protected]
wrote:

Even using FFI, you’ll need to define the wrapping between C constant to
Ruby constant, and using Ruby’s API, you’ll be define_const-ing every one of
those, then putting them into an array. Just define them in the Ruby
yourself, there isn’t a faster way to do it.

I was afraid of that :frowning: The docs are a bit on the sketchy side too. I
guess some sort of code generation would be the way to go then (swig
seems like overkill). Will give cgenerator a look.

martin

Martin DeMello wrote:

Using it (or want to use it!) from the ruby side to generate a hash of
constant => name and use that to unpack and display flag settings.

If all you want is this data structure in ruby, you could parse the
source (gccxml?), and then you have a pure ruby library that defines
this hash. Define the parse->xml>rb part as a rake task if you need to
keep up to date with the source files.

On Sat, Jun 13, 2009 at 2:52 AM, Joel
VanderWerf[email protected] wrote:

Cgenerator will work, and might be a good idea if the source (the list of
values and strings) changes from time to time, and you would rather have
your program regenerate the ruby extension (and data structure)
automatically.

Here’s how it works, assuming your source files are flag.c and flag.h:

Thanks, that looks great. I have to say, I’m a bit disappointed in the
state of ruby/C integration - another basic thing I missed is the
ability to define a typemap and then have a ruby class
serialised/deserialised to a C struct. (Is that what Cgenerator’s
CShadow is all about?). Are these not common use cases?

martin

Martin DeMello wrote:

martin
Cgenerator will work, and might be a good idea if the source (the list
of values and strings) changes from time to time, and you would rather
have your program regenerate the ruby extension (and data structure)
automatically.

Here’s how it works, assuming your source files are flag.c and flag.h:

$ ls
flag.c flag.h flag.rb
$ cat flag.h
struct flag_str {
unsigned int val;
const char *str;
};

extern struct flag_str extent_flags[];

typedef enum {
FIEMAP_EXTENT_LAST = 1,
FIEMAP_EXTENT_UNKNOWN,
FIEMAP_EXTENT_DELALLOC,
FIEMAP_EXTENT_NO_BYPASS,
FIEMAP_EXTENT_SECONDARY,
FIEMAP_EXTENT_NET,
FIEMAP_EXTENT_DATA_COMPRESSED,
FIEMAP_EXTENT_DATA_ENCRYPTED,
FIEMAP_EXTENT_NOT_ALIGNED,
FIEMAP_EXTENT_DATA_INLINE,
FIEMAP_EXTENT_DATA_TAIL,
FIEMAP_EXTENT_UNWRITTEN,
FIEMAP_EXTENT_MERGED
} FIEMAP_EXTENT;

$ cat flag.c
#include “flag.h”

#ifndef NULL
#define NULL 0
#endif

struct flag_str extent_flags[] = {
{ FIEMAP_EXTENT_LAST, “last” },
{ FIEMAP_EXTENT_UNKNOWN, “unkown” },
{ FIEMAP_EXTENT_DELALLOC, “delalloc” },
{ FIEMAP_EXTENT_NO_BYPASS, “no_bypass” },
{ FIEMAP_EXTENT_SECONDARY, “secondary” },
{ FIEMAP_EXTENT_NET, “net” },
{ FIEMAP_EXTENT_DATA_COMPRESSED, “data_compressed” },
{ FIEMAP_EXTENT_DATA_ENCRYPTED, “data_encrypted” },
{ FIEMAP_EXTENT_NOT_ALIGNED, “not_aligned” },
{ FIEMAP_EXTENT_DATA_INLINE, “data_inline” },
{ FIEMAP_EXTENT_DATA_TAIL, “data_tail” },
{ FIEMAP_EXTENT_UNWRITTEN, “unwritten” },
{ FIEMAP_EXTENT_MERGED, “merged” },
{ 0, NULL },
};

$ cat flag.rb
#!/usr/bin/env ruby

require ‘cgen/cgen’
require ‘fileutils’

module Flag; end

Generate the extension source code.

lib = CGenerator::Library.new “flag_lib”
lib.include “flag.h”

lib.define_c_singleton_method(Flag, :extent_flags).instance_eval {

no args

body %{
VALUE a;
struct flag_str *pfs;

 a = rb_ary_new();
 for (pfs = extent_flags; pfs->val; pfs++) {
   rb_ary_push(a,
     rb_ary_new3(2,
       INT2NUM(pfs->val),
       rb_str_new2(pfs->str)
   ));
 }

}
returns “a”
}

Normally, this isn’t needed, but we need to set up the external files

as symlinks in this dir before calling commit.

FileUtils.mkdir_p(“flag_lib”)

Put links to sources where they will be found.

Dir.chdir “flag_lib” do
%w{ flag.h flag.c }.each do |file|
FileUtils.ln_s("…/#{file}", file) rescue nil
end
end

Write and build

lib.commit

Use the library

Flag::EXTENT_FLAGS = Hash[*Flag.extent_flags.flatten]
p Flag::EXTENT_FLAGS

$ ruby flag.rb
{5=>“secondary”, 11=>“data_tail”, 6=>“net”, 12=>“unwritten”, 1=>“last”,
7=>“data_compressed”, 13=>“merged”, 2=>“unkown”, 8=>“data_encrypted”,
3=>“delalloc”, 9=>“not_aligned”, 4=>“no_bypass”, 10=>“data_inline”}

On Mon, Jun 15, 2009 at 5:13 AM, Martin DeMello
[email protected]wrote:

Thanks, that looks great. I have to say, I’m a bit disappointed in the
state of ruby/C integration - another basic thing I missed is the
ability to define a typemap and then have a ruby class
serialised/deserialised to a C struct. (Is that what Cgenerator’s
CShadow is all about?). Are these not common use cases?

martin

Could you give an example of how this would work? I’m having a hard time
picturing how such a thing would be possible to have done automagically.
There has to be something somewhere that defines how a Ruby class would
map
to said C struct.

Jason

Martin DeMello wrote:

another basic thing I missed is the
ability to define a typemap and then have a ruby class
serialised/deserialised to a C struct. (Is that what Cgenerator’s
CShadow is all about?). Are these not common use cases?

Would this be like swig in reverse? Input a ruby class with type
annotations, output C header, accessors, etc? I’d like to do that for
the special case of BitStruct classes (fixed-length, packed fields,
usually numeric or character, stored in a ruby String using
pack/unpack), so that once you define a network protocol or other binary
format in ruby, you can share your code with C programmers (and via swig
with other languages). That seems feasible, and I did actually do this
with a predecessor of BitStruct, using cgen. But how would you handle a
more general case, like a class with a hash or array attribute?

CShadow is similar, but instead of inheriting from String like
BitStruct, its instances are T_DATA (ruby objects with a blob that can
only be accessed from C). Unlike a BitStruct, this blob is not intended
for sending outside of the ruby+C world. For example, a “self” pointer
is automatically added at the beginning of the blob.

Including CShadow in your class lets you define the structure of the
T_DATA object in ruby, in terms of accessors of various types. When your
code runs, it uses cgenerator to build an extension that takes care of
all the boiler plate functions: mark, free, accessors with type
checking/conversion, alloc, init, inspect, marshal, and yaml. Plus it
handles inheritance and sets up introspection methods. [See example
below.]

CShadow also gives you tools to distribute source code in separate .c
and .h files, so that when you make a minor change in the definitions
embedded in ruby, you minimize how much needs to be recompiled. This
works well enough that even if the generated code is a few Mb, you can
let cgenerator do its full generate/make cycle every time you run your
program, without noticing much delay.

You can add C functions to a CShadow class inline in ruby source just as
with cgenerator. But CShadow only supports a few kinds of attributes
“out of the box”: ruby object references, native scalar data of various
sizes, and native pointers to char, double, array of double, etc. You
can add new attr types with a little work, but it’s far from automatic.

These are just two special case solutions. I’m not seeing what a general
solution would look like. What kinds of classes/attributes are there in
your use cases?


require ‘cgen/cshadow’

class Parent
include CShadow

shadow_attr_accessor :ruby_str => String # type-checked VALUE type
shadow_attr_accessor :c_int => “int c_int”
end

class Child < Parent
shadow_attr_accessor :c_str => “char *c_str”
shadow_attr_accessor :obj => Object # VALUE type
end

Parent.commit

we’re done adding attrs and methods, so make.

x = Child.new
x.ruby_str = “foo”
x.c_str = “bar”
x.obj = [1,2,3]
x.c_int = 3

p x

CShadow.allow_yaml
y x

END

Output:

#<Child:0xb79716cc ruby_str=“foo”, c_int=3, c_str=“bar”, obj=[1, 2, 3]>
— !path.berkeley.edu,2006/cshadow:Child
ruby_str: foo
c_int: 3
c_str: bar
obj:

  • 1
  • 2
  • 3

On Mon, Jun 15, 2009 at 7:13 PM, Jason R.[email protected]
wrote:

Could you give an example of how this would work? I’m having a hard time
picturing how such a thing would be possible to have done automagically.
There has to be something somewhere that defines how a Ruby class would map
to said C struct.

Yes, I was simply looking for something like

struct foo {
int bar;
char *baz;
}

cFoo = rb_define_class(‘cFoo’)
rb_map_struct(cFoo, foo)
rb_accessor(cFoo, ‘int’, ‘bar’, bar)
rb_accessor(cFoo, ‘str’, ‘bar’, bar)

VALUE rb_function() {
foo* a = some_function_call()
return rb_deserialize_data(foo)
}

and from ruby

foo = rb_function
foo.bar
foo.baz

martin

On Mon, Jun 15, 2009 at 11:00 PM, Joel
VanderWerf[email protected] wrote:

Martin DeMello wrote:

another basic thing I missed is the
ability to define a typemap and then have a ruby class
serialised/deserialised to a C struct. (Is that what Cgenerator’s
CShadow is all about?). Are these not common use cases?

Would this be like swig in reverse? Input a ruby class with type
annotations, output C header, accessors, etc?

I actually meant like a simplified swig - you have a C struct
(arbitrary nesting of structs and arrays, actually), and you have a
ruby class that you define a typemap for and then have it
serialise/deserialise. Basically you know that you can mirror C
structures in ruby with a bit of gruntwork, all you need is a tool to
do the gruntwork for you

I’d like to do that for the
special case of BitStruct classes (fixed-length, packed fields, usually
numeric or character, stored in a ruby String using pack/unpack), so that
once you define a network protocol or other binary format in ruby, you can
share your code with C programmers (and via swig with other languages). That
seems feasible, and I did actually do this with a predecessor of BitStruct,
using cgen. But how would you handle a more general case, like a class with
a hash or array attribute?

Network protocols were one of the use cases I was thinking of, except
that you usually see them defined in C. I wanted the ability to use
the structs from ruby, without going through C the way
data_wrap_struct makes you do. Not asking for magic, just the ability
to make the common case trivial.

Including CShadow in your class lets you define the structure of the T_DATA
object in ruby, in terms of accessors of various types. When your code runs,
it uses cgenerator to build an extension that takes care of all the boiler
plate functions: mark, free, accessors with type checking/conversion, alloc,
init, inspect, marshal, and yaml. Plus it handles inheritance and sets up
introspection methods. [See example below.]

Okay, that’s most of what I was looking for :slight_smile: If it can convert an
array of structs into a ruby array, that’s all of what I’m looking
for. Need to look through the CGenerator docs some more - great piece
of work!

martin

Martin DeMello wrote:

If it can convert an array of structs into a ruby array, that’s all of what I’m looking for.

How is the length stored? A separate field? Or is it fixed length?

On Mon, Jun 15, 2009 at 11:11 AM, Martin DeMello
[email protected]wrote:

VALUE rb_function() {
martin

You are attacking your problem from the wrong end. The problem is that C
doesn’t have reflection so there is no way the rb_map_struct concept
would
ever work. C won’t tell you what the members are so you can’t map it to
anything.

However, working in the other direction - you could use the C
preprocessor
to build both your structs and the appropriate C code to define your
ruby
class you want for each struct. It’s not pretty but it does work - have
a
look here for a nice defintion of what you are looking for

John

On Tue, Jun 16, 2009 at 1:55 AM, Joel
VanderWerf[email protected] wrote:

Martin DeMello wrote:

If it can convert an array of structs into a ruby array, that’s all of
what I’m looking for.

How is the length stored? A separate field? Or is it fixed length?

In true C style, you need to pass in a pointer in which the length is
returned :slight_smile: I’m not thinking of anything magical; I’d expect to pass
in the length as a parameter.

martin