[ANN] Net::LDAP 0.0.3 released, adds TLS encryption

We’re pleased to announce version 0.0.3 of Net::LDAP, the first
pure-Ruby LDAP library. Net::LDAP intends to be a feature-complete
LDAP client which can access as much as possible of the functionality
of the most-used LDAP server implementations. This library does
not wrap any existing native-code LDAP libraries, creates no
Ruby extensions, and has no dependencies external to Ruby.

Version 0.0.3 adds support for encrypted communications to LDAP servers.
There is a new optional parameter for Net::LDAP#new and Net::LDAP#open
that allows you to specify encryption characteristics. Here’s a quick
example:

require ‘net/ldap’
ldap = Net::LDAP.new(
:host => “an_ip_address”,
:port => 636,
:auth => {:method => :simple, :username => “mickey”, :password =>
“mouse” },
:encryption => {:method => :simple_tls}
)
ldap.bind or raise “bind failed”
ldap.search( … )

etc, etc.

This release supports simple TLS encryption with no client or server
validation. Future versions will add support for the STARTTLS control,
and for certificate validation. Additional parameters will appear to
support these options.

Net::LDAP encryption requires Ruby’s openssl library. We’re not
quite sure what happens when this library is present but the underlying
OpenSSL libraries are missing or not configured appropriately,
especially on back versions of Ruby. If anyone encounters problems
using encryption in Net::LDAP, please let us know and give us the
details of your platform and Ruby build info.

Thanks to Garett Shulman for helping to test the new code.

If anyone wants to contribute suggestions, insights or (especially)
code, please email me at garbagecat10 … … gmail.com.

Francis C. wrote:

We’re pleased to announce version 0.0.3 of Net::LDAP, the first
pure-Ruby LDAP library. Net::LDAP intends to be a feature-complete
LDAP client which can access as much as possible of the functionality
of the most-used LDAP server implementations. This library does
not wrap any existing native-code LDAP libraries, creates no
Ruby extensions, and has no dependencies external to Ruby.

This is great. But why is the use of modify deprecated? If I have a
group membership attribute with, say, a hundred values, and I just want
to add or remove one value, isn’t it more efficient and safer (in the
presence of concurrent updates) to use modify?

thanks

Justin

Justin F. wrote:

But why is the use of modify deprecated? If I have a
group membership attribute with, say, a hundred values, and I just want
to add or remove one value, isn’t it more efficient and safer (in the
presence of concurrent updates) to use modify?

thanks

Justin

From the Rdoc entry for Net::LDAP#modify:

DEPRECATED - Please use add_attribute, replace_attribute, or
delete_attribute

#modify was deprecated because we thought the API was too complex, so we
broke it up into three new functions. Under the covers, the three new
attribute-modification APIs are just sugar on top of the original
#modify, which might become private someday. The current Rdoc commentary
is all still correct and valid.

To your specific point: “safer” is dependent on the LDAP server
implementation. The vast majority of them don’t have the kind of
concurrency you would expect in an RDBMS. As far as “efficient” goes,
you’re not really going to get much help in your particular example,
because LDAP doesn’t let you simply modify one attribute out of a
hundred. You will end up deleting all of them and re-adding them with
your changes reflected in the new set. That’s built into the LDAP wire
protocol.

It’s been my experience that a lot of people expect LDAP servers to
behave like database engines. Some of them in fact are implemented on
top of DBMSs (Tivoli’s TDS is a good example, and is in fact one of the
better implementations, although it’s too damned expensive so I never
recommend it). But the LDAP protocol itself works against RDBMS-like
behaviors. LDAP is quite a different animal, and I think this is an
underappreciated fact.

Justin F. wrote:

In our application, maintenance of user
organisations, users, profiles, and functions is done under
dual-operator control (one administrator makes a change, another
authorises it) in DB2, and a subset of authorised data needs to be
replicated to TDS. I’m prototyping in Ruby; the real implementation will
be in Java.

TDS is already in DB2 under the covers, so you’re doing a bit of a
two-step to maintain AZ tables in DB2 and partially propagate them to
TDS.

To answer your question as you posed it: I’m going to assume you’re
writing a trigger of some kind to modify your TDS data when your AZ
tables change. LDAP defines an “attribute” as an attribute-name and a
set of values. The set of values may be of any cardinality, including
zero, depending on the constraints of your schema. When you search for
an entry via LDAP, the protocol will return to you all of the values
for each attribute you request. (This could be hundreds of values, of
course.) You can modify the set locally using Net::LDAP APIs. When you
update the attribute, all of the values, including the ones you didn’t
modify, get sent back to the LDAP server.

Read the RDoc for Net::LDAP#modify carefully to understand the
distinctive semantics that LDAP defines for these operations: they are
well-defined, precise, rather complex, somewhat unintuitive, and quite
unlike typical database operations.

Is this a problem? Well, how often do your AZ rules change? TDS is a
fairly powerful LDAP implementation and can handle a fair amount of
flux. (If you had to do this in Active Directory, I’d strongly advise
you not even to try.) But the fact that you are running TDS tells me
that you are probably a large company beholden somewhow to IBM. (Maybe
they told you that they wouldn’t support Websphere unless you
implemented TDS?) So you’ll have to experiment and see if this will
work. I will say that Net::LDAP is going to be at least as fast as JNDI
on the wire, so if you find that your prototype is slow, don’t expect it
to be magically faster when you go to Java.

Now I’ll try to answer your question on somewhat different terms: why
are you using an LDAP directory for authorization in the first place?
You should be using it for authentication and identity-management,
nothing more. There are other, far more effective approaches for AZ.

Feel free to send me a private email if you want to discuss your
application in more detail.

Francis C. wrote:

From the Rdoc entry for Net::LDAP#modify:
To your specific point: “safer” is dependent on the LDAP server
implementation. The vast majority of them don’t have the kind of
concurrency you would expect in an RDBMS.

I appreciate that, and am trying to design accordingly.

As far as “efficient” goes,

you’re not really going to get much help in your particular example,
because LDAP doesn’t let you simply modify one attribute out of a
hundred. You will end up deleting all of them and re-adding them with
your changes reflected in the new set. That’s built into the LDAP wire
protocol.

I didn’t know that. So the interpretation of the modifications is done
on the client side? And if I add one value to an attribute all the
values are brought to the client, one is added, and the new set is sent
back?

Take a scenario where I find the groups containing a user, and want to
remove the user. I haven’t retrieved the full membership of those
groups, but I can express removing the use as a modification on each
group. What is the right way to implement this?

It’s been my experience that a lot of people expect LDAP servers to
behave like database engines. Some of them in fact are implemented on
top of DBMSs (Tivoli’s TDS is a good example, and is in fact one of the
better implementations, although it’s too damned expensive so I never
recommend it). But the LDAP protocol itself works against RDBMS-like
behaviors. LDAP is quite a different animal, and I think this is an
underappreciated fact.

I am aware that it is a different animal, and I’m trying to make sure we
use it appropriately. In our application, maintenance of user
organisations, users, profiles, and functions is done under
dual-operator control (one administrator makes a change, another
authorises it) in DB2, and a subset of authorised data needs to be
replicated to TDS. I’m prototyping in Ruby; the real implementation will
be in Java.

Thanks very much for your help.

Justin

Francis C. wrote:

TDS.
Yes, indeed.

To answer your question as you posed it: I’m going to assume you’re
writing a trigger of some kind to modify your TDS data when your AZ
tables change. LDAP defines an “attribute” as an attribute-name and a
set of values. The set of values may be of any cardinality, including
zero, depending on the constraints of your schema. When you search for
an entry via LDAP, the protocol will return to you all of the values
for each attribute you request. (This could be hundreds of values, of
course.) You can modify the set locally using Net::LDAP APIs. When you
update the attribute, all of the values, including the ones you didn’t
modify, get sent back to the LDAP server.

All our changes are timestamped in the application’s database (and all
deletions are logical deletes), so we can replicate frequently,
focussing on what has changed since the last successful replication, and
retry on failure.

There is much more data being maintained (under dual-operator control)
than the organisation/user/profile/function data that is needed for
authentication/authorization.

After replying earlier, I had a quick look at RFC1777

http://www.ietf.org/rfc/rfc1777.txt

and the description of modification (section 4.4) suggests that
additions and deletions of attribute values can be done incrementally
over the wire.

(Your Net::LDAP add_attribute method will add values to those already
existing, but your delete_attribute method deletes all values.)

Read the RDoc for Net::LDAP#modify carefully to understand the
distinctive semantics that LDAP defines for these operations: they are
well-defined, precise, rather complex, somewhat unintuitive, and quite
unlike typical database operations.

Yes. Given that it is not possible to wrap multiple operations in a
transaction, it seems valuable to be able to make fine-grained
modifications. Otherwise someone else could update a multi-valued
attribute between our read and our write. (We should be the only source
of updates to the directory content, but I’d like to make the solution
as robust as possible.) If I can just say “remove this user’s DN from
the members of the Admin group”, that seems better (and exactly what the
protocol was designed to be able to do).

Is this a problem? Well, how often do your AZ rules change? TDS is a
fairly powerful LDAP implementation and can handle a fair amount of
flux. (If you had to do this in Active Directory, I’d strongly advise
you not even to try.) But the fact that you are running TDS tells me
that you are probably a large company beholden somewhow to IBM. (Maybe
they told you that they wouldn’t support Websphere unless you
implemented TDS?) So you’ll have to experiment and see if this will
work. I will say that Net::LDAP is going to be at least as fast as JNDI
on the wire, so if you find that your prototype is slow, don’t expect it
to be magically faster when you go to Java.

Our client is a large company, and IBM runs their applications. They
(IBM) are pushing hard for the use of TDS with WebSphere
container-managed authentication and authorization, and they have the
client’s ear. This is happening in mid-project; we already had all the
authentication and authorization working against DB2.

Now I’ll try to answer your question on somewhat different terms: why
are you using an LDAP directory for authorization in the first place?
You should be using it for authentication and identity-management,
nothing more. There are other, far more effective approaches for AZ.

Feel free to send me a private email if you want to discuss your
application in more detail.

I’ll take you up on that (but not over the weekend) - thank you very
much indeed.

Justin

Justin F. wrote:

After replying earlier, I had a quick look at RFC1777

As a point of interest, the current LDAP standard is 2251, which
obsoletes 1777.

and the description of modification (section 4.4) suggests that
additions and deletions of attribute values can be done incrementally
over the wire.

I’m going to re-read that section very carefully and change Net::LDAP
accordingly, if possible. But my recollection is that LDAP uses the term
“attribute value” to refer to the whole (possibly multi-valued) set, not
the individual items in the set. Net::LDAP was implemented strictly from
the RFC, and I don’t remember seeing any protocol support for changing
individual items in an attribute value.

Otherwise someone else could update a multi-valued
attribute between our read and our write.

They sure can.

Our client is a large company, and IBM runs their applications. They
(IBM) are pushing hard for the use of TDS with WebSphere
container-managed authentication and authorization, and they have the
client’s ear. This is happening in mid-project; we already had all the
authentication and authorization working against DB2.

Everyone know Microsoft plays dirty, but they are amateurs compared to
IBM.

Francis C. wrote:

Justin F. wrote:

After replying earlier, I had a quick look at RFC1777

As a point of interest, the current LDAP standard is 2251, which
obsoletes 1777.

Yes, 1777 was for LDAP version 2, but it was the first one I found, and
I thought it would be good enough for checking the semantics of modify.

[RFC4511 (June 2006) will obsolete 2251 (when taken together with 4510,
4512, and 4513).]

and the description of modification (section 4.4) suggests that
additions and deletions of attribute values can be done incrementally
over the wire.

I’m going to re-read that section very carefully and change Net::LDAP
accordingly, if possible. But my recollection is that LDAP uses the term
“attribute value” to refer to the whole (possibly multi-valued) set, not
the individual items in the set. Net::LDAP was implemented strictly from
the RFC, and I don’t remember seeing any protocol support for changing
individual items in an attribute value.

The descriptions of add and delete semantics are:

 add: add values listed to the given attribute, creating
 the attribute if necessary;

 delete: delete values listed from the given attribute,
 removing the entire attribute if no values are listed, or
 if all current values of the attribute are listed for
 deletion;

In 2251, there is an additional stipulation that deletion of specific
values is only permitted if the attribute type defines an equality match
filter.

Note that the RFC requires the modifications to take place in the order
in which they were specified, and that the overall operation is atomic.
RFC1777 used “should”: “The entire list of entry modifications should be
performed in the order they are listed, as a single atomic operation.”
but 2251 uses “MUST”.

Everyone know Microsoft plays dirty, but they are amateurs compared to
IBM.

:slight_smile:

regards

Justin

Francis C. wrote:

I’m going to re-read that section very carefully and change Net::LDAP
accordingly, if possible. But my recollection is that LDAP uses the term
“attribute value” to refer to the whole (possibly multi-valued) set, not
the individual items in the set. Net::LDAP was implemented strictly from
the RFC, and I don’t remember seeing any protocol support for changing
individual items in an attribute value.

I believe you have implemented it correctly, and from playing with
Net::LDAP in irb it seems straightforward (using modify) to delete a
specific attribute value, leaving other values of the same attribute
intact.

pp entries[0]
#<Net::LDAP::Entry:0x2d2c048
@myhash=
{:dn=>[“cn=user_admin,ou=roles,o=xyz,c=uk”],
:cn=>[“user_admin”],
:objectclass=>[“top”, “groupOfUniqueNames”],
:description=>[“User Administration”],
:uniquemember=>
[“cn=SuperUser,ou=profiles,o=xyz,c=uk”,
“cn=Administrator,ou=profiles,o=xyz,c=uk”]}>
=> nil
ldap.modify :dn => “cn=user_admin,ou=roles,o=xyz,c=uk”,

  • :operations => [[:delete,
  •                'uniquemember',
    
  •                'cn=Administrator,ou=profiles,o=xyz,c=uk']]
    

=> true

ldap.search(:base => ‘ou=roles,o=xyz,c=uk’,

  • :filter => Net::LDAP::Filter.eq(‘cn’, ‘user_admin’)) {|e| pp e}; nil
    #<Net::LDAP::Entry:0x2f63bd8
    @myhash=
    {:dn=>[“cn=user_admin,ou=roles,o=xyz,c=uk”],
    :cn=>[“user_admin”],
    :objectclass=>[“top”, “groupOfUniqueNames”],
    :description=>[“User Administration”],
    :uniquemember=>[“cn=SuperUser,ou=profiles,o=xyz,c=uk”]}>
    => nil

Given the ordering and atomicity specified for the modify operation, it
would be a pity to continue to deprecate it.

Thanks again for an excellent library. I hope it gets coverage in the
next edition of “Enterprise Integration with Ruby”. Maybe ActiveLDAP
should move over to a pure Ruby implementation, too.

regards

Justin

P.S. I’ll be offline now until Sunday evening.

Justin F. wrote:

Given the ordering and atomicity specified for the modify operation, it
would be a pity to continue to deprecate it.

I just remembered the other reason why I deprecated #modify: because of
LDAP’s lack of transaction semantics. If you use Net::LDAP#modify to
execute a chain of operations, perhaps on multiple attributes, and one
of them fails in the middle, the preceding ones DON’T get rolled back.
If this happens, most LDAP servers won’t give you a very useful error
message, either. So I think it makes more sense to just code such
operations discretely and handle any errors on a per-operation basis.

But we should keep #modify public, in case there are servers out there
that observe some kind of nonstandard transaction semantics or
concurrency guarantees, and I’m glad you pointed that out. I’ll change
the Rdoc to reflect that.

Francis C. wrote:

If this happens, most LDAP servers won’t give you a very useful error
message, either. So I think it makes more sense to just code such
operations discretely and handle any errors on a per-operation basis.

But we should keep #modify public, in case there are servers out there
that observe some kind of nonstandard transaction semantics or
concurrency guarantees, and I’m glad you pointed that out. I’ll change
the Rdoc to reflect that.

I haven’t quite got offline yet: I’m just packing to go down to the
coast (UK South coast) for the rest of the weekend.

RFC2251 (and 4511) are quite clear that the sequence of modifications
should be atomic:

- modification: A list of modifications to be performed on the 

entry.
The entire list of entry modifications MUST be performed
in the order they are listed, as a single atomic operation. While
individual modifications may violate the directory schema, the
resulting entry after the entire list of modifications is
performed
MUST conform to the requirements of the directory schema.

[snip]

The server will return to the client a single Modify Response
indicating either the successful completion of the DIT modification,
or the reason that the modification failed. Note that due to the
requirement for atomicity in applying the list of modifications in
the Modify Request, the client may expect that no modifications of
the DIT have been performed if the Modify Response received 

indicates
any sort of error, and that all requested modifications have been
performed if the Modify Response indicates successful completion of
the Modify Operation. If the connection fails, whether the
modification occurred or not is indeterminate.

Thanks again (and I shall get in touch off-list about the other points
you have made)

Justin

Justin F. wrote:

I believe you have implemented it correctly, and from playing with
Net::LDAP in irb it seems straightforward (using modify) to delete a
specific attribute value, leaving other values of the same attribute
intact.

Thanks for checking. I found basically the same thing. Net::LDAP follows
the spec closely enough to enable the backend to do the right thing, so
no changes needed. And my re-reading of both 1777 and 2251 supports your
view of what the backend should be doing, so I’m glad we went through
the exercise.

Given the ordering and atomicity specified for the modify operation, it
would be a pity to continue to deprecate it.

That’s a very, very good point. I still dislike the API of #modify, too
complicated, not Ruby-esque enough. Maybe there’s a syntactic way (a
block?) to specify sequences of add, changes, and deletes so they will
go in a single LDAP Modify request and achieve the atomicity provided by
the protocol. Does anyone have any suggestions? Patches gladly accepted.

Thanks again for an excellent library. I hope it gets coverage in the
next edition of “Enterprise Integration with Ruby”. Maybe ActiveLDAP
should move over to a pure Ruby implementation, too.

I corresponded briefly with Will Drewry about that and looked through
his code. The native LDAP API is scattered pretty extensively through
one of the modules in ActiveLDAP so it’s not a five-minute job. Anyone
out there willing to help me do it?

Justin F. wrote:

RFC2251 (and 4511) are quite clear that the sequence of modifications
should be atomic:

- modification: A list of modifications to be performed on the 

entry.
The entire list of entry modifications MUST be performed
in the order they are listed, as a single atomic operation. While
individual modifications may violate the directory schema, the
resulting entry after the entire list of modifications is
performed
MUST conform to the requirements of the directory schema.

[snip]

The server will return to the client a single Modify Response
indicating either the successful completion of the DIT modification,
or the reason that the modification failed. Note that due to the
requirement for atomicity in applying the list of modifications in
the Modify Request, the client may expect that no modifications of
the DIT have been performed if the Modify Response received 

indicates
any sort of error, and that all requested modifications have been
performed if the Modify Response indicates successful completion of
the Modify Operation. If the connection fails, whether the
modification occurred or not is indeterminate.

I’ve specifically tested this against several different well-known LDAP
servers and found that most of them violate the atomicity requirement. I
haven’t tested TDS for this. If you can do such a test against TDS, I’d
be most interested in hearing the result.

Justin, you’d better hurry down to the coast while there is still some
weekend left ;-). But thanks for all the attention you’re paying to
this.

Francis C. wrote:

I’ve specifically tested this against several different well-known LDAP
servers and found that most of them violate the atomicity requirement. I
haven’t tested TDS for this. If you can do such a test against TDS, I’d
be most interested in hearing the result.

That’s depressing. Are those servers certified for LDAP compliance? If
you could send me a test I’ll see what I can do with respect to TDS.

Justin, you’d better hurry down to the coast while there is still some
weekend left ;-). But thanks for all the attention you’re paying to
this.

:slight_smile: I enjoyed a bit of walking, swimming, and a few pints in the country
pubs.

regards

Justin

(answering again, as my reply 16 hours ago hasn’t come back to me via
the mailing list yet)

Francis C. wrote:

I once had to implement an LDAP server from scratch for a particular
application (still in production), and query-performance is about 30
times better than OpenLDAP.

OK. I was thinking of the Open Group certification:

Open Brand

I found more links relating to LDAP certification here:

http://www.freelists.org/archives/ldapdata/05-2005/msg00000.html

Do you have a straightforward test for atomicity that you could share?

thanks

Justin

Justin F. wrote:

Francis C. wrote:

I’ve specifically tested this against several different well-known LDAP
servers and found that most of them violate the atomicity requirement. I
haven’t tested TDS for this. If you can do such a test against TDS, I’d
be most interested in hearing the result.

That’s depressing. Are those servers certified for LDAP compliance? If
you could send me a test I’ll see what I can do with respect to TDS.

Justin, you’d better hurry down to the coast while there is still some
weekend left ;-). But thanks for all the attention you’re paying to
this.

:slight_smile: I enjoyed a bit of walking, swimming, and a few pints in the country
pubs.

regards

Justin

LOL! What is LDAP compliance certification? In practice there are only a
few directory servers in wide use (a lot of iPlanet, some TDS and some
Oracle in enterprises, OpenLDAP on all the scruffy Linux boxes, and
ActiveDirectory everywhere). So I’d have to suppose as a practical
matter that LDAP compliance comes down to what A/D does: an extremely
low common denominator, extremely bad performance, and full of awful
bugs that will probably never be fixed.

OpenLDAP is probably the most compliant LDAP server I’ve seen, although
it too has quirks. And it actually deserves its reputation for slowness.
I once had to implement an LDAP server from scratch for a particular
application (still in production), and query-performance is about 30
times better than OpenLDAP.

Francis C. wrote:

Justin F. wrote:

Francis C. wrote:

I’ve specifically tested this against several different well-known LDAP
servers and found that most of them violate the atomicity requirement. I
haven’t tested TDS for this. If you can do such a test against TDS, I’d
be most interested in hearing the result.
That’s depressing. Are those servers certified for LDAP compliance? If
you could send me a test I’ll see what I can do with respect to TDS.

I once had to implement an LDAP server from scratch for a particular
application (still in production), and query-performance is about 30
times better than OpenLDAP.

So you don’t rate the certification by the Open Group:

Open Brand

I’m interested to know why, and I’d still like you to send me a test for
atomicity.

regards

Justin