Hello Reader, here is this month’s iRODS news and developments!

If you’re facing an issue with iRODS you’re not sure how to solve, please do drop me a line; if I’ve come across a solution or seen something relevant elsewhere, I’ll do my best to let you know. Or just drop me a mail to say ‘Hi’. Always nice to hear from people, particularly in these pandemic times!

I’d love your thoughts and feedback on how this newsletter could be better for you.

News

iRODS UGM 2022

It happened! I was there, and so were a number of you (welcome, new subscribers!). You can see us all in front of a Belgian castle in this tweet.

I wrote a round-up on my website;

The unofficial iRODS documentation project

At the end of the UGM I made a call during a lighting talk, for people interested in writing system-admin and best practice focussed documentation for the community and quite a few people said they would, so work has started with a GitHub Repository holding a MK Docs site. It’s early days right now - we don’t even have a theme at the time I write this, or a deployed site, but contributions are welcome!

Consultancy services available!

I had so much fun at the UGM talking to everyone I completely forgot to mention that I am available for some small consultancy if that would be useful - initially around training, but do drop me a mail if you think I might be able to help.

More details on my website​

iRODS Python Client Library 1.1.4 is released

​1.1.4​

Main Repository Activity

Open Issues

​Investigate CMake Presets​

"This CMake feature seems like a very good way to capture various build configurations. It may also save developers from having to write build scripts whose main purpose is to set various build options."

​Refactor / Modernize main server logic (rodsServer.cpp, etc.)​

"The iRODS server was originally written in C around 20 years ago. We would like to refactor the core server software with C++ to use modern, high-level techniques for purposes of maintainability and performance as well as extensibility."

​PAM auth does not work after upgrading to 4.3.0-1​

'If you're using icommands, replace the use of "PAM" in your irods_environment.json file with "pam_password".'

One fr the ‘when upgrading to 4.3’ list!

​iRODS server​

A nice bit of developer documentation here, might steal it for the community project…

​Add lazy evaluation of format arguments to 4.3 logging API​

Enhancement request: “There are situations where the logger can be passed an object that is expensive to construct or copy and never actually logged.

By the time the logging API determines that the object will be ignored, it is too late. We've already paid the price of construction.
The logging API needs to provide a way to skip this."

​irsync -r fails to sync two collections with false SYS_COPY_LEN_ERR​

To be clear, this is between two collections not between a file system ad a collection, and on 4.2.8. I am told that >= 4.2.9 might clear up a lot of this, as I’ve found unusable (along with) in 4.2.8.

​remote( ) microservice does not honour INST_NAME​

​Dynamic _pre PEPs do not propagate error message to client like Static ones do​

​Plugin build hooks require locally built and packages​

​Call msiDataObjRead using python rule engine​

"The python rule engine plugin build hook (https://github.com/irods/irods_rule_engine_plugin_python/blob/dff2042e8119643bb03395bef8dc790b10c22461/irods_consortium_continuous_integration_build_hook.py#L76) requires built iRODS packages to be supplied in order to build the plugin. It would be nice to be able to use the officially released packages from packages.irods.org or core-dev.irods.org for building with the build hook."

Not sure if this is just 4.3.x or affects earlier as well? Probably the latter.

​setup_irods.py Assumes default_resource_name to be present in server_config​

When configuring 4.3.0 with the json_configuration_file option, setup_irods.py should be present at the top level, but not in. However, setup_irods.py looks for it in both locations. The issue also has a link to an MR provide to patch setup_irods.py.

​Enhancement Request: Add index on checksum to r_data_main table ​

Searching data names is fast, whereas searching for checksums is slow. The reporter suggests adding an index at database creation/upgrade time.

I like the idea of pre-setup, but I’m not sure about this specifically. There is database overhead to maintain an index, so if you are writing lots of checksums (e.g. you have msiDataObjChksum in an upload rule), you might not want this as the default, and it can be added later by a DBA et al.

​Investigate twice-changing error type in servers 4.2.8 → 4.3.0​

"in data_obj_test.py , PRC reports three different possible errors test_create_from_invalid_path__250 based on changes in server version only.
ex.SYS_INVALID_INPUT_PARAM: prior to 4.2.9
ex.CAT_UNKNOWN_COLLECTION: from 4.2.9 - 4.2.11"+" ( current tip of 4.2.stable at 3989081 and icommands=c5f663c8bb1f932a8e66131cb8d4ccd39a7e98da)
ex.SYS_INVALID_FILE_PATH: beginning in 4.3.0"

​iput numThreads​

This seems to be an issue of site specific configuration - of one server not being able to resolve, or connect to another server on the higher ports required for multithreaded large file transfer.

​metadata.hpp contains definitions that should be in a cpp file somewhere.​

"metadata.hpp is full of function definitions that should be moved to a .cpp file and replaced with declarations. In its current state, including the header more than once in a single library or executable will result in linker errors due to overlapping symbol definitions."

4.3.0

Closed Issues

Closed on - 2022-07-05 15:13:51 Caught exception in migrate_delay_server​

"Further investigation has revealed that this only happens with 4.3.0.0 of the audit rule engine plugin installed and configured according to the training slide deck.
As this does not happen with a default configuration or with packages built from tip of main, I think this is actually an issue with the audit rule plugin and was fixed with irods/irods_rule_engine_plugin_audit_amqp#99."

Closed on - 2022-06-29 15:42:29 Reference custom plugin rules in custom rule play book​

"In my custom.re rule playbook I would like to call a rule defined in a custom plugin (a .so file) which is defined in my server conf with a particular instance_name. "

Good discussion in how to use a custom plugin in your configuration.

Closed on - 2022-06-23 18:30:42 GitHub Actions: git224-core no longer available for centos:7 builds​

Closed on - 2022-06-23 18:30:29 Scope error for constants in irods/apiNumber.h​

4.3.0

"in irods/apiNumber.h, constants are defined which should be static. This matters when the file is included (for example, through irods/rodsClient.h) in multiple C source files, leading to a clash at link time."

Closed on - 2022-06-18 10:21:35 iget -r cannot be combined with -n​

They can, but, you probably shouldn’t.

Python iRODS Client Activity

Open Issues

​irods.exception.SYS_NO_API_PRIV when creating group for group admin​

• iRODS 4.3.0
• Python client 1.1.4

​Temp variable ‘pam_password’ easily confused with entity in core code.​

Closed Issues

Closed on - 2022-07-07 13:12:54 efficient way to get size of a specified collection ​

"If there is an efficient way to get the size of all the data objects under specified collection(maybe recursive) like linux du -s
Thanks a lot."

and a sample bit of code to do this from the issue!

total_size_in_bytes = 0 n_data_objs = 0 collection = session.collections.get('/myZone/home/myUser/home/testCollection') for info in collection.walk( ): n_data_objs += len( info[2] ) total_size_in_bytes += sum( d.size for d in info[2] )

Closed on - 2022-06-29 20:35:55 client library emits confusing message on initial use​

Possibly fixed in 1.1.4?

Closed on - 2022-06-24 13:15:41 noisy socket exit message​

To be fixed in 1.1.5? In the meantime, you can pull the master branch.

Closed on - 2022-06-23 19:22:53 allow direct calls to set via object handles​

To be fixed in 1.1.5? In the meantime, you can pull the master branch.

Closed on - 2022-06-24 13:15:09 PAM authentication fails when there is an ‘=’ character in the password​

To be fixed in 1.1.5? In the meantime, you can pull the master branch.

Closed on - 2022-05-20 12:47:30 Atomic metadata operations don’t work with the group permission​

"In case a user has own access only provided by the group permission, then this user cannot add metadata via the atomic operations."

fixed in later versions of iRODS;

"By the way, our core dev team seems to have fixed the problem on the way to upcoming releases post 4.2.11 :
4.2.12
4.3.0"

Closed on - 2022-06-29 20:35:43 fcntl not available on Windows​

To be fixed in 1.1.5? In the meantime, you can pull the master branch.

Closed on - 2022-02-19 03:42:30 failing tests need revision for minor relative-path concerns​

icommands Activity

Closed Issues

Closed on - 2022-06-22 02:56:21 iput numThreads​

"closing as duplicate of conversation at irods/irods#6447"

YODA Activity

Open Issues

​[BUG] License metadata is not sent to DataCite​

If you think someone else would appreciate this newsletter, they can sign up at https://theresource.metadata.school/​

Four Yaks were shaved in the making of this newsletter.