Hello Reader, here is this month's iRODS news and developments!
Before we dive in, a reminder that I still have some very limited availability for consultancy. If you have a project that could use my expertise, please get in touch. 1:1 training, architecture review, development, and troubleshooting are all possible.
Find out more at my work in progress website
News
Its likely that I will be moving this newsletter away from kit (was ConvertKit) to some other provider (probably Buttondown) between now and the next newsletter. Buttondown is a much better fit for my needs, and is also much cheaper, but kit.com's massive increase in thier fees for small newsletters like mine is what's really pushing me to make the change.
No A.I. Summary this month. Do you prefer this months slightly more verbose style, or the more concise A.I. version? Let me know by replying to this email...
My Blog Posts
I've started posting regularly on LinkedIn. Here are my most recent posts:
RENCI Mastodon Toots
- releases in a month!? https://irods.org/2025/08/irods-development-update- august-2025/ #irods #python #pam #java #http #s3 #internsCard: iRODS
- library compatible with iRODS 5, adds support for the PAM Interactive authentication scheme, improves support for groupadmins, and adds new features for GenQuery1. [https://github.com/irods/python- irodsclient/releases/tag/v3.2.0](https://github.com/irods/python- irodsclient/releases/tag/v3.2.0) #irods #python #pam #datamanagementCard: Release v3.2.0 · irods/python-irodsclient Description: This release makes the library compatible with iRODS 5, adds support for the PAM Interactive authentication scheme, improves support for groupadmins, and adds new features for GenQuery1. With this ...
- with iRODS 5, improves compatibility with earlier versions of iRODS, expands support for targeting replicas for I/O operations, improves write performance to data objects, and removed OAuth client mode (supporting only protected resource mode now). Dockerfiles for EL8 and EL9 have been added to make building packages easier. [https://github.com/irods/irods_client_http_api/rel eases/tag/0.6.0](https://github.com/irods/irods_client_http_api/releases/tag/0 .6.0) #irods #cpp #http #oidc #datamanagementCard: Release 0.6.0 · irods/irods_client_http_api Description: This release makes the HTTP API compatible with iRODS 5, improves compatibility with earlier versions of iRODS, expands support for targeting replicas for I/O operations, and improves write perform...
- compatible with iRODS 5 and adds support for Presigned URLs. The server also performs validation of the configuration file on startup. [https://github.com /irods/irods_client_s3_api/releases/tag/0.4.0](https://github.com/irods/irods_ client_s3_api/releases/tag/0.4.0) #irods #s3 #datamanagementCard: Release 0.4.0 · irods/irods_client_s3_api Description: This release makes the S3 API compatible with iRODS 5 and adds support for Presigned URLs. The server also performs validation of the configuration file on startup. Changed Update default JSON sch...
GitHub Activity
Main Repository (irods/irods)
Open Issues – irods/irods
- State: open Labels: bug Description: Version, OS and Version iRODS 4.3.4 and 5.0.1 Ubuntu 24.04 LTS ### What did you try to do? Get or set a grid configuration variable using
iadmin with wrong parameters. ### Expected behavior If a failure occurs when getting or setting grid configuration variables, iadmin returns a non-zero return value. ### Observed Behavior (including steps to reproduce, if applicable) iadmin seems to always have a return value of 0 with these subcommands, irrespective of whether the operation succeeded. For example: irods@provider:~$ iadmin get_grid_configuration doesnotexist doesnotexist Failed to get grid configuration for namespace [doesnotexist] and option [doesnotexist] [ec=-808000] irods@provider:~$ echo $? 0 irods@provider:~$ iadmin set_grid_configuration doesnotexist doesnotexist something Failed to set grid configuration for namespace [doesnotexist] and option [doesnotexist] [ec=-808000] irods@provider:~$ echo $? 0
- State: open Labels: question Description: ## Question
Standard configuration of a catalog service consumer results in server_config.json not containing a database stanza. This raises the question - Does adding an empty database stanza affect the server's behavior?
For example:
{"plugin_configuration":{"database":{}}}
- State: open Labels: bug, testing Description: observed while testing irods/irods_testing_environment#275. It feels unrelated to the testing environment PR.
--- IrodsSession: icommand executed by [alice#tempZone] [itouch /tempZone/home/alice/2025-09-10Z18:24:03-- irods-testing-od16feb3/all_keyword_dataobj] --- Assert Command: itouch /tempZone/home/alice/2025-09-10Z18:24:03--irods- testing-od16feb3/all_keyword_dataobj Expecting EMPTY: [''] stdout: | stderr: | Output found --- IrodsSession: icommand executed by [otherrods#tempZone] [iadmin mkresc resc_all_keyword unixfilesystem 8a380f9d7422:/tmp/irods-testing- tw5b9fpq/resc_all_keyword_vault] --- Assert Command: iadmin mkresc resc_all_keyword unixfilesystem 8a380f9d7422:/tmp/irods-testing-tw5b9fpq/resc_all_keyword_vault Expecting STDOUT: ['resc_all_keyword'] stdout: | Creating resource: | Name: "resc_all_keyword" | Type: "unixfilesystem" | Host: "8a380f9d7422" | Path: "/tmp/irods-testing-tw5b9fpq/resc_all_keyword_vault" | Context: "" stderr: | Output found --- IrodsSession: icommand executed by [alice#tempZone] [irepl -R resc_all_keyword /tempZone/home/alice/2025-09-10Z18:24:03--irods- testing-od16feb3/all_keyword_dataobj] --- Assert Command: irepl -R resc_all_keyword /tempZone/home/alice/2025-09-10Z18:24:03--irods-testing- od16feb3/all_keyword_dataobj Expecting EMPTY: [''] stdout: | stderr: | Output found --- IrodsSession: icommand executed by [alice#tempZone] [iput -f /tmp/irods-test-irsync/all_keyword_dataobj /tempZone/home/alice/2025-09-10Z18:24:03--irods-testing- od16feb3/all_keyword_dataobj] --- Assert Command: iput -f /tmp/irods-test-irsync/all_keyword_dataobj /tempZone/home/alice/2025-09-10Z18:24:03--irods-testing- od16feb3/all_keyword_dataobj Expecting EMPTY: [''] stdout: | stderr: | Output found --- IrodsSession: icommand executed by [alice#tempZone] [iquest %s select DATA_REPL_STATUS where DATA_NAME = 'all_keyword_dataobj' and DATA_REPL_NUM = '1'] --- --- IrodsSession: icommand executed by [alice#tempZone] [irsync -a /tmp/irods-test-irsync/all_keyword_dataobj i:/tempZone/home/alice/2025-09-10Z18:24:03--irods-testing- od16feb3/all_keyword_dataobj] --- Assert Command: irsync -a /tmp/irods- test-irsync/all_keyword_dataobj i:/tempZone/home/alice/2025-09-10Z18:24:03--irods-testing- od16feb3/all_keyword_dataobj Expecting EMPTY: [''] stdout: | stderr: | remote addresses: 172.18.0.5 ERROR: rsyncUtil: rsync error for /tempZone/home/alice/2025-09-10Z18:24:03--irods-testing- od16feb3/all_keyword_dataobj status = -1803000 HIERARCHY_ERROR Unexpected output on stderr FAILED TESTING ASSERTION
- State: open Labels: bug, testing Description: observed while testing irods/irods_testing_environment#275. It feels unrelated to the testing environment PR.
- State: open Labels: bug Description: not update the usage on that coordinating resource. > > Just one comment to add following discussion about this. > > Q. Should physical quotas be allowed on coordinating resources? > A. We feel it should. Whether that's a good use of the feature is up to the admin. > > --- > > The question is motivated by a resource hierarchy containing a replication resource and three unixfilesystem resources. If a quota is attached to the replication resource and a user stores a file there, the replication resource will replicate it to its children. > > What should the quota report for the replication resource? > Should the user's file cost 3x its size due to there being three physical replicas? > > This situation led us to the answer of ... the quota system is about physical storage and should count all replicas. The quota system is a tool and admins are given freedom over where it is applied. Originally posted by @korydraughn in #4089
- State: open Labels: enhancement Description: ### New Feature / Functionality
https://github.com/irods/irods/blob/15a76b6a30f8749430452be9c3145bfb6b2dd46f/server/core/src/rodsConnect.cpp#L61-L85
rcatZoneHint is only used in a call to getRcatHost and is passed in to a const char* parameter:
https://github.com/irods/irods/blob/15a76b6a30f8749430452be9c3145bfb6b2dd46f/server/core/include/irods/rodsConnect.h#L115-L117
This function is only exposed to applications linking against the server package, and making the type const char* from char* should be safe.
If not, we should make a new function with the proposed parameter type and deprecate the existing one.
Please add the C/C++ deprecation attribute to applyAllRules and related functions. This allows the compiler to detect use of the deprecated functions and stop the compilation.
This is in support of #8660.
- State: open Labels: documentation, deprecation / removal Description: https://github.com/irods/irods/issues/2502
- State: open Labels: bug, consortium-member, compatibility Description: likely caused by the following line. https://github.com/irods/irods_cli ent_icommands/blob/84d053292251e4be7dd727cb2b2ba42b62af11b6/src/imiscsvr info.cpp#L90 The packing instructions changed in 4.3.4 making the icommand incompatible with older versions of iRODS. Notice
certinfo. https://github.com/irods/irods/blob/3f6524069c0eac458fc26808b7095a414581 a28e/lib/api/include/irods/getMiscSvrInfo.h#L19-L24 This can be fixed by using a custom/compatible packing instruction and procApiRequest directly. Originally reported in #8638.
- State: open Labels: documentation Description: Need to think about whether any plugins need to be released alongside this server release.
- State: open Labels: enhancement Description: ### New Feature / Functionality
irsync has never considered partial-file changes when deciding what data to send/sync. It is file-level only and will send the entire source file, even if the destination file is already mostly the same. It is noted here that iput, iget, and icp are all similar use cases (perhaps with one side of the sync simply being empty / not-yet-in-existence).
librsync provides signature, delta, and patch operations to detect, describe, and apply changes to keep a source and destination file in sync / identical.
First places to investigate adding this capability include irsync, putUtil, and/or syncUtil. Initial planning suggests storage resource plugins will not need to know about this addition - they would still handle talking to storage the same way they do now, but this is not yet certain.
Not sure whether whole-file (high-level) or streaming (low-level) calls would be best for this work:
Success here would reduce the amount of data for iRODS to transfer, theoretically increasing 'throughput' significantly.
Other client libraries (Python, Java, Go, etc.) may also implement the decided algorithm / solution, but that would be future work, separate from this issue.
- State: open Labels: bug, consortium-member, compatibility Description: ### New Feature / Functionality The iRODS 5.0.1 icommands are not compatible with an iRODS 4.2.x server. As the server reports its version, the icommands could use such information to proactively notify the user of the incompatibility with a more gentle message.
Examples of current output:
iinit (user has populated irods_environment.json with institute-suggested values (pam)):
$ iinit
Connecting as XXX to XXX:1247 ...
Level 0: Error occurred while authenticating user [XXX] [PLUGIN_ERROR_MISSING_SHARED_OBJECT: [-] /usr/include/irods/irods_load_plugin.hpp:448:error irods::load_plugin(PluginType *&, const std::string &, const std::string &, const std::string &, const Ts &...) [PluginType = irods::authentication::authentication_base, Ts =
$
imiscsvrinfo:
$ imiscsvrinfo
Segmentation fault (core dumped)
$
- State: open Labels: bug Description: working on #8583, the following command was executed to see what would happen. (from https://github.com/irods/irods_client_icommands/pull/584# discussion_r2301702891)
igroupadmin mkuser alice "" remoteZone This resulted in the following output. Level 0 message: DEBUG: remote addresses: 127.0.0.1 ERROR: rcUserAdmin failed with error -806000 CAT_SQL_ERR Level 0: DEBUG: remoteZone was created by running the following. iadmin mkzone remoteZone remote This was observed against a 5.0.1 server. Please investigate.
- State: open Labels: bug, build / tools Description: ## Bug Report
iRODS Version, OS and Version
4.1.0+ (all the way up to tip of main)
What did you try to do?
Use irods::check_sent_sid in a target which includes irods_client_server_negotiation.hpp and does not link against the server package. Here's the declaration in question: https://github.com/irods/irods/blob/ad59ae9c52331ddbc19dac71bc05461b00f46885/lib/core/include/irods/irods_client_server_negotiation.hpp#L99-L100
Expected behavior
Because the function is declared as available for use, I expect to be able to use it in my client application.
Observed Behavior (including steps to reproduce, if applicable)
Okay, I didn't actually try this, but I can see that it will fail to link because check_sent_sid is only defined in server/core/src/irods_server_negotiation.cpp: https://github.com/irods/irods/blob/ad59ae9c52331ddbc19dac71bc05461b00f46885/server/core/src/irods_server_negotiation.cpp#L19-L67
Here is its only (known) usage: https://github.com/irods/irods/blob/ad59ae9c52331ddbc19dac71bc05461b00f46885/server/core/src/irods_server_negotiation.cpp#L149-L154
We should remove this from the client library header since it is only available for targets linking against the server package. It may not require external linkage at all since it's only used in one compilation unit.
Open question: Can this happen in a minor or even a patch release?
iRODS Server Logs (if applicable)
N/A
- State: open Labels: enhancement, question, performance, refactor Description: ## Enhancement
We know that renaming/moving large collections can be very expensive in iRODS. They are also destructive. This issue presents how we can potentially improve the performance of this operation through the introduction of a new database table. But first, we need to review how path information is stored in the database.
The relevant pieces of information within the database are:
-
r_coll_main
coll_id
coll_name: The logical path of a collection
-
r_data_main
coll_id: Reference to the parent collection in r_coll_main
data_name: The filename - i.e. basename of the logical path
data_path: The absolute physical path of a replica in storage
One of the reasons for why iRODS is slow when it comes to renames/moves is due to it needing to potentially update all entries in the database which store a copy of the physical path.
Proposal / Idea
What if we modeled the physical path information as a tree-like structure in the database?
First, we'd replace the data_path column in r_data_main with a single new column, phy_id.
This column links to a table holding decomposed physical paths within a storage system. Let's call the table r_phy_main. It contains the following columns:
id: The unique id of the row. This is what phy_id in r_data_main uses to form a link.
pid: The id of the parent. This points to another row within this table.
type: The type of the path element - e.g. file (f), directory (d), etc.
path_element: A single string representing a single path element. Not required to be unique.
Here's an example of what the data within this table would look like.
idpidtypepath_element10000d/10011000dhome10021000dtrash10031001dother10041001drods10051004ffoo10061004fbar
Notes
- A
pid of 0 indicates the root of a path hierarchy
- All physical paths are derived recursively via database CTEs or are readily available via a database view
path_element is purely for display, no need to compare strings
- A tuple of (
pid, path_element) protects the catalog from introducing duplicate entries within the same directory
Believed Outcome
- All rename/move operations become significantly faster
- Flexibility - path manipulation becomes easier and can be accomplished in more than one way
r_resc_main would need to be updated to use an id from r_phy_main as its root element
- The database consumes less storage because path elements are reusable
Questions
- Where does this fall apart?
- What are we forgetting?
- Will this design be an issue for production deployments?
- Is there a better approach?
- What is the blast radius for such a change?
- Is there other information worth storing in this table?
Comments
From @trel:
All rename/move operations become significantly faster
becomes
All rename/move operations become a single update (O(1)), rather than 'the number of descendents' updates (O(N))
From me:
While this approach could work in theory, it's possible that with a large enough table, the performance of a CTE could be poor due to its recursive nature among other things.
We'd need to test this to be sure though.
Knowing the usage patterns of physical paths would help in determining whether this is viable as well. The important questions are, IMO:
How often do users need to know the physical path of multiple replicas?Where are large numbers of physical paths encountered most often? Which operations?Can GenQuery2 help with improving performance?
- State: open Labels: bug Description: working on quotas support for the HTTP API, I discovered that recalculating physical quotas can produce the wrong totals. It appears to be related to timing and database transactions. The problem is easily reproducible with this PR, https://github.com/irods/irods_client_http_api/pull/457. Just running the quotas test,
test_quotas_endpoint, reveals the issue. We thought that this was due to the connection pool, but it's not. The issue still appeared with the connection pool disabled. This means it should be reproducible with icommands, which it is. Below is a shell script which demonstrates the problem. If the script does not reproduce the issue for you, try the HTTP API PR previously mentioned. bash #! /bin/bash dobj=quotas.txt # init quotas and show stat irm -f "$dobj" iadmin sgq public total 10 iadmin cu iadmin lq # create a data object and update the quota totals echo -n ok | istream write "$dobj" iadmin cu iadmin lq # overwrite the data object and update the quota totals echo -n This puts the quota in violation | istream write "$dobj" iadmin cu # show the data object cannot be written to even if we're truncating it echo -n This puts the quota in violation | istream write "$dobj" # show the quota totals iadmin lq # <== THIS LINE PRESENTS INCORRECT TOTALS SOMETIMES Here's one run showing incorrect results. Notice quota_over: 24 in the final block. That should report quota_over: 22. The last invocation of iadmin cu does not appear to calculate the correct totals. Per resource quotas (if any) for users/groups: No rows found Global (total usage) quotas (if any) for users/groups: user_name: public zone_name: tempZone quota_limit: 10 quota_over: -10 modify_ts: 2025-08-20.17:22:58 Per resource quotas (if any) for users/groups: No rows found Global (total usage) quotas (if any) for users/groups: user_name: public zone_name: tempZone quota_limit: 10 quota_over: -8 modify_ts: 2025-08-20.17:22:59 Error: Cannot open data object. Per resource quotas (if any) for users/groups: No rows found Global (total usage) quotas (if any) for users/groups: user_name: public zone_name: tempZone quota_limit: 10 quota_over: 24 modify_ts: 2025-08-20.17:22:59 Adding calls to sleep within the HTTP API's quota tests resolves the issue, but we need to understand why things do not work as intended when the sequence of operations are under a controlled environment. This was observed when running the HTTP API against 4.3.4 and 5.0.1. Please investigate.
- State: open Labels: enhancement Description: ## Enhancement
The 2025 Roadmap states that the stable plugins are to be absorbed in a future version of iRODS. From https://irods.org/roadmap/#2025:
Absorb stable plugins into serverMetadata Guard rule engine pluginLogical Quotas rule engine pluginStorage Tiering capability pluginS3 resource plugin
As part of this effort, we want to investigate an alternative design for Logical Quotas. This new design would be modeled after the physical quotas implementation which has existed in iRODS for quite some time. This new design would use a new database table for storing quota information and drop use of PEPs and (possibly) metadata.
The motivation behind this design change is that we think all of the design decisions for the Logical Quotas REP align with the design of the physical quotas system. Here are the primary things which motivated this idea:
- The current REP implementation cannot guarantee 100% accuracy of totals - i.e. there will be drift
-
The current REP implementation provides a
recalculate-totals command because of drift- It is recommended that admins run this command periodically to keep totals accurate
- The current REP implementation allows write operations one violation of the quota constraints, to avoid performance issues
A huge motivator behind this idea is the fact that it greatly simplifies the porting of Logical Quotas.
We will provide updates here as things progress. We want Logical Quotas in the server. How that looks will depend on what we learn through investigating this idea.
iRODS Version, OS and Version
iRODS 4.3.4 and 5.0.1 Ubuntu 24.04 LTS
What did you try to do?
Show contents of a replica of a data object, while referring to a resource that does not have a replica of this data object.
Commands to reproduce this situation:
$ mkdir /tmp/resc1 /tmp/resc2
$ iadmin mkresc resc1 unixfilesystem provider.local:/tmp/resc1
Creating resource:
Name: "resc1"
Type: "unixfilesystem"
Host: "provider.local"
Path: "/tmp/resc1"
Context: ""
$ iadmin mkresc resc2 unixfilesystem provider.local:/tmp/resc2
Creating resource:
Name: "resc2"
Type: "unixfilesystem"
Host: "provider.local"
Path: "/tmp/resc2"
Context: ""
$ echo test > test.txt
$ iput -R resc1 test.txt
$ istream read -R resc2 test.txt
Expected behavior
Istream shows an error message about not being able to find the replica, like iget would in this situation. Example iget output:
$ iget -R resc2 test.txt download.test.txt
remote addresses: 192.168.121.216 ERROR: getUtil: get error for download.test.txt status = -168000 SYS_REPLICA_INACCESSIBLE
Level 0: hierarchy descending from requested resource name does not have a replica or the replica is inaccessible at this time [path=[/testZone/home/rods/test.txt], requested resource name=[resc2], resolved hierarchy=[resc1]]
$ echo $?
3
Observed Behavior (including steps to reproduce, if applicable)
No error message. Istream shows data from other replica:
$ istream read -R resc2 test.txt
test
$ echo $?
0
iRODS Server Logs (if applicable)
I don't see anything relevant here.
- State: open Labels: bug, consortium-member Description: ## Bug Report
iRODS Version, OS and Version
irods5.0.0/1
What did you try to do?
We encountered a search issue while using the PRC client to search for metadata attached by msiAssociateKeyValuePairsToObj. After debugging, we discovered that iRODS clients handle malformed characters, such as (ASCII 4), null bytes, and other non-printable characters, differently when adding and searching for metadata.
Therefore I tried to add/search as metadata via various irods client.
Expected behavior
I expected all iRODS clients to behave consistently, either by successfully handling these characters or by returning an error.
Observed Behavior (including steps to reproduce, if applicable)
When querying malformed metadata, non-cpp client like the PRC and goclient fail, but iquest and ils handle it as an empty text. In metadata add operations, the PRC fails, whereas msiAssociateKeyValuePairsToObj successfully adds the records, which are then visible in the catalog.
add this character like below:
root@e863e5fc7aac:/# cat rule.r
import irods_types
def add_metadata(callback, obj_name, attribute, value):
"""Adds key value pair to an object"""
keyValPair = callback.msiString2KeyValPair(
attribute + "=" + value, irods_types.KeyValPair()
)["arguments"][1]
callback.msiAssociateKeyValuePairsToObj(keyValPair, obj_name, "-d")
def main(rule_args, callback, rei):
callback.writeLine('serverLog', 'Execute command to print out hello')
object_path = "/u0137480_irods433/home/rods/hello.txt"
#add_metadata(callback, object_path, 'atr', 'val')
add_metadata(callback, object_path, '', 'hhA-test_mustafa')
add_metadata(callback, object_path, '@\b', '2025-05-17T12:14:48+02:00')
INPUT null
OUTPUT ruleExecOut
root@e863e5fc7aac:/# irule -r irods_rule_engine_plugin-python-instance -F rule.r
try with the PRC like below:
root@e863e5fc7aac:/# cat test.py
import os, os.path
from irods.session import iRODSSession
from irods.models import DataObject, DataObjectMeta, Collection, CollectionMeta
env_file = os.getenv('IRODS_ENVIRONMENT_FILE', os.path.expanduser('~/.irods/irods_environment.json'))
with iRODSSession(irods_env_file=env_file) as session:
pass
obj = session.data_objects.get("/icts/home/operator/test1.txt")
obj.metadata.add('', 'hhA-test_mustafa')
#results = session.query(DataObjectMeta)
#for r in results:
# print(r[DataObjectMeta.name], r[DataObjectMeta.value])
root@e863e5fc7aac:/# python3 test.py
Traceback (most recent call last):
File "//test.py", line 11, in
obj.metadata.add('', 'hhA-test_mustafa')
File "/usr/local/lib/python3.12/dist-packages/irods/meta.py", line 152, in add
self._reset_metadata()
File "/usr/local/lib/python3.12/dist-packages/irods/meta.py", line 105, in _reset_metadata
self._meta = self._manager.get(self._model_cls, self._path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/irods/manager/metadata_manager.py", line 86, in get
results = self.sess.query(*columns).filter(*conditions).all()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/irods/query.py", line 233, in all
result_set = self.execute()
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/irods/query.py", line 220, in execute
results = result_message.get_main_message(GenQueryResponse)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/irods/message/__init__.py", line 448, in get_main_message
msg.unpack(ET().fromstring(self.msg))
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.12/xml/etree/ElementTree.py", line 1335, in XML
parser.feed(text)
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 14, column 7
to query with iCommands:
root@e863e5fc7aac:/# imeta ls -d /icts/home/operator/test.txt
AVUs defined for dataObj /icts/home/operator/test.txt:
attribute: @
value: 2025-05-17T12:14:48+02:00
units:
----
attribute:
value: hhA-test_mustafa
units:
root@e863e5fc7aac:/# iquest " select META_DATA_ATTR_NAME, META_DATA_ATTR_VALUE where COLL_NAME like '/icts/home/operator%'"
META_DATA_ATTR_NAME = @
META_DATA_ATTR_VALUE = 2025-05-17T12:14:48+02:00
------------------------------------------------------------
META_DATA_ATTR_NAME =
META_DATA_ATTR_VALUE = hhA-test_mustafa
------------------------------------------------------------
PRC query:
root@e863e5fc7aac:/# cat test.py import os, os.path from irods.session import iRODSSession
env_file = os.getenv('IRODS_ENVIRONMENT_FILE', os.path.expanduser('~/.irods/irods_environment.json')) with iRODSSession(irods_env_file=env_file) as session: pass
obj = session.data_objects.get("/icts/home/operator/test.txt") print(obj.metadata)
root@e863e5fc7aac:/# python3 test.py Traceback (most recent call last): File "//test.py", line 9, in print(obj.metadata) ^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/irods/data_object.py", line 85, in metadata self._meta = iRODSMetaCollection( ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/irods/meta.py", line 102, in init self._reset_metadata() File "/usr/local/lib/python3.12/dist-packages/irods/meta.py", line 105, in _reset_metadata self._meta = self._manager.get(self._model_cls, self._path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/irods/manager/metadata_manager.py", line 86, in get results = self.sess.query(*columns).filter(*conditions).all() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/irods/query.py", line 233, in all result_set = self.execute() ^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/irods/query.py", line 220, in execute results = result_message.get_main_message(GenQueryResponse) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/irods/message/init.py", line 448, in get_main_message msg.unpack(ET().fromstring(self.msg)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/xml/etree/ElementTree.py", line 1335, in XML parser.feed(text) xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 15, column 8
gocommand query:
root@e863e5fc7aac:/# gocmd lsmeta /icts/home/operator/test.txt Unexpected error! Error Trace:
- failed to list meta for path "/icts/home/operator/test.txt": github.com/cyverse/gocommands/cmd/subcmd.(*LsMetaCommand).listMetaForPath /github/workspace/cmd/subcmd/lsmeta.go:116
- failed to receive a data object metadata query result message: github.com/cyverse/go-irodsclient/irods/fs.ListDataObjectMeta /go/pkg/mod/github.com/cyverse/go-irodsclient@v0.17.3-0.20250507211308-472d5351cc1e/irods/fs/data_object.go:1373
- failed to parse response message: github.com/cyverse/go-irodsclient/irods/connection.(*IRODSConnection).RequestWithTrackerCallBack /go/pkg/mod/github.com/cyverse/go-irodsclient@v0.17.3-0.20250507211308-472d5351cc1e/irods/connection/request_response.go:93
- failed to parse a response message: github.com/cyverse/go-irodsclient/irods/connection.(*IRODSConnection).getResponse /go/pkg/mod/github.com/cyverse/go-irodsclient@v0.17.3-0.20250507211308-472d5351cc1e/irods/connection/request_response.go:266
- failed to get irods message from message body: github.com/cyverse/go-irodsclient/irods/message.(*IRODSMessageQueryResponse).FromMessage /go/pkg/mod/github.com/cyverse/go-irodsclient@v0.17.3-0.20250507211308-472d5351cc1e/irods/message/query_response.go:61
- failed to unmarshal xml to irods message: github.com/cyverse/go-irodsclient/irods/message.(*IRODSMessageQueryResponse).FromBytes /go/pkg/mod/github.com/cyverse/go-irodsclient@v0.17.3-0.20250507211308-472d5351cc1e/irods/message/query_response.go:45
- XML syntax error on line 15: illegal character code U+0008
### iRODS Server Logs (if applicable)
- [`msiAssociateKeyValuePairsToObj(keyValPair` returns `CAT_INVALID_ARGUMENT` in irods4.3.4](https://github.com/irods/irods/issues/8625)
**State:** open
**Labels:** bug, consortium-member
**Description:** ## Bug Report
### iRODS Version, OS and Version
irods4.3.3, irods4.3.3, irods5.0.0/1
### What did you try to do?
I used `msiAssociateKeyValuePairsToObj(keyValPair)` to attach metadata to an iRODS data object.
### Expected behavior
I expected that the provided key-value metadata would be successfully attached.
### Observed Behavior (including steps to reproduce, if applicable)
My attempts failed (in 4.3.4), and errors were logged on both the client side and in the iRODS logs.
**steps to reproduce**
in a rule (.r) file:
import irods_types
def add_metadata(callback, obj_name, attribute, value): """Adds key value pair to an object""" keyValPair = callback.msiString2KeyValPair( attribute + "=" + value, irods_types.KeyValPair() )["arguments"][1] callback.msiAssociateKeyValuePairsToObj(keyValPair, obj_name, "-d")
def main(rule_args, callback, rei): callback.writeLine('serverLog', 'Execute command to print out hello') object_path = "/u0137480/home/rods/hello.txt" add_metadata(callback, object_path, 'atr', 'val')
INPUT null OUTPUT ruleExecOut
[irods@dev-u0137480 irods]$ irule -r irods_rule_engine_plugin-python-instance -F rule.r remote addresses: 10.0.2.100 ERROR: rcExecMyRule error. status = -1828000 RULE_ENGINE_ERROR
or in core.py:
import irods_types
def pep_api_data_obj_put_post(rule_args, callback, rei):
obj_path = str(rule_args[2].objPath) callback.writeLine('serverLog', obj_path) add_metadata(callback, obj_path, "atr", "val")
def add_metadata(callback, obj_name, attribute, value): """Adds key value pair to an object""" keyValPair = callback.msiString2KeyValPair( attribute + "=" + value, irods_types.KeyValPair() )["arguments"][1] callback.msiAssociateKeyValuePairsToObj(keyValPair, obj_name, "-d")
def convert_unicode_hyphen(path): """Converts '‐' the non-breaking hyphen.""" unicode_hyphen = "‐" regular_hyphen = "-" return path.replace(unicode_hyphen, regular_hyphen)
[irods@dev-u0137480 irods]$ iput myfile remote addresses: 10.0.2.100 ERROR: putUtil: put error for /u0137480/home/rods/myfile, status = -816000 status = -816000 CAT_INVALID_ARGUMENT
### iRODS Server Logs (if applicable)
{"processname": "irods", "groupname": "irods", "pid": "134", "channel": "stdout", "irods": {"log_category": "legacy", "log_level": "info", "log_message": "writeLine: inString = Execute command to print out hello\n", "request_api_name": "EXEC_MY_RULE_AN", "request_api_number": 625, "request_api_version": "d", "request_client_user": "rods", "request_host": "10.0.2.100", "request_proxy_user": "rods", "request_release_version": "rods4.3.4", "server_host": "dev-u0137480.irods.t.icts.kuleuven.be", "server_pid": 31223, "server_timestamp": "2025-08-20T13:16:08.621Z", "server_type": "agent", "server_zone": "u0137480"}} {"processname": "irods", "groupname": "irods", "pid": "134", "channel": "stdout", "irods": {"log_category": "legacy", "log_level": "info", "log_message": "rsModAVUMetadata: rcModAVUMetadata failed", "request_api_name": "EXEC_MY_RULE_AN", "request_api_number": 625, "request_api_version": "d", "request_client_user": "rods", "request_host": "10.0.2.100", "request_proxy_user": "rods", "request_release_version": "rods4.3.4", "server_host": "dev-u0137480.irods.t.icts.kuleuven.be", "server_pid": 31223, "server_timestamp": "2025-08-20T13:16:08.621Z", "server_type": "agent", "server_zone": "u0137480"}} {"processname": "irods", "groupname": "irods", "pid": "134", "channel": "stdout", "irods": {"log_category": "rule_engine", "log_level": "error", "log_message": "caught python exception", "python_exception": "Traceback (most recent call last):\n File "", line 14, in main\n File "", line 8, in add_metadata\nRuntimeError: [iRods__Error__Code:-816000] [-]\t/irods_source/server/re/include/irods/irods_re_plugin.hpp:333:irods::error irods::dynamic_operation_execution_managerboost::any &) : status [CAT_INVALID_ARGUMENT] errno [] -- message [exec_microservice_adapter failed]\n\n\n", "request_api_name": "EXEC_MY_RULE_AN", "request_api_number": 625, "request_api_version": "d", "request_client_user": "rods", "request_host": "10.0.2.100", "request_proxy_user": "rods", "request_release_version": "rods4.3.4", "rule_engine_plugin": "python", "server_host": "dev-u0137480.irods.t.icts.kuleuven.be", "server_pid": 31223, "server_timestamp": "2025-08-20T13:16:08.622Z", "server_type": "agent", "server_zone": "u0137480"}} {"processname": "irods", "groupname": "irods", "pid": "134", "channel": "stdout", "irods": {"log_category": "api", "log_level": "error", "log_message": "rsExecMyRule: -1828000, [-]\t/irods_plugin_source/src/main.cpp:851:irods::error exec_rule_text(const irods::default_re_ctx &, const std::string &, msParamArray_t *, const std::string &, irods::callback) : status [RULE_ENGINE_ERROR] errno [] -- message [irods_rule_engine_plugin_python::irods::error exec_rule_text(const irods::default_re_ctx &, const std::string &, msParamArray_t *, const std::string &, irods::callback) Caught Python exception.\nTraceback (most recent call last):\n File "", line 14, in main\n File "", line 8, in add_metadata\nRuntimeError: [iRods__Error__Code:-816000] [-]\t/irods_source/server/re/include/irods/irods_re_plugin.hpp:333:irods::error irods::dynamic_operation_execution_managerboost::any &) : status [CAT_INVALID_ARGUMENT] errno [] -- message [exec_microservice_adapter failed]\n\n\n]\n\n", "request_api_name": "EXEC_MY_RULE_AN", "request_api_number": 625, "request_api_version": "d", "request_client_user": "rods", "request_host": "10.0.2.100", "request_proxy_user": "rods", "request_release_version": "rods4.3.4", "server_host": "dev-u0137480.irods.t.icts.kuleuven.be", "server_pid": 31223, "server_timestamp": "2025-08-20T13:16:08.622Z", "server_type": "agent", "server_zone": "u0137480"}}
- [Review core.re to determine if there are options which should be treated as zone-wide options](https://github.com/irods/irods/issues/8624)
**State:** open
**Labels:** enhancement, documentation, configuration
**Description:**
is motivated by work on quotas for the HTTP API. See https://github.com/irod
s/irods_client_http_api/pull/457#discussion_r2283290021.
- [GenQuery1 queries for resource quotas do not use the same query conditions for `iadmin lq`](https://github.com/irods/irods/issues/8622)
**State:** open
**Labels:** question, bug, refactor
**Description:**
GenQuery1 queries used for reporting resource quotas do not share the
same query conditions. One uses `QUOTA_RESC_ID != '0'` while the other
uses `QUOTA_RESC_NAME != '0'`. Seems to me that they should both use
`QUOTA_RESC_ID`. - https://github.com/irods/irods_client_icommands/blob/
5ca283625595555ba0a505d38efe0e398aa2ba0d/src/iadmin.cpp#L946 - https://g
ithub.com/irods/irods_client_icommands/blob/5ca283625595555ba0a505d38efe
0e398aa2ba0d/src/iadmin.cpp#L970 I think the queries lead to the same
outcome, but we should sync them to avoid confusion later.
- [Document or change the effects of disabling and re-enabling group quotas](https://github.com/irods/irods/issues/8618)
**State:** open
**Labels:** question, documentation, consortium-member
**Description:**
that has been updated due to changes in the data, and then re-enabling
that same group quota does not result in the calculated quota values
being reset. Here's an example: ```bash $ # Show we do not have any
quotas set right now. $ iadmin lq Per resource quotas (if any) for
users/groups: No rows found Global (total usage) quotas (if any) for
users/groups: No rows found $ # Set a global quota for the public group.
$ iadmin sgq public total 100 $ # Show we have no data. $ ils
/tempZone/home/rods: $ # Create a data object and show that the quota
information $ # has not been updated. $ echo -n hello iRODS | istream
write file.txt $ iadmin lq Per resource quotas (if any) for
users/groups: No rows found Global (total usage) quotas (if any) for
users/groups: user_name: public zone_name: tempZone quota_limit: 100
quota_over: -100 modify_ts: 2025-08-18.15:43:52 $ # Update the quota
information and show the results. $ iadmin cu $ iadmin lq Per resource
quotas (if any) for users/groups: No rows found Global (total usage)
quotas (if any) for users/groups: user_name: public zone_name: tempZone
quota_limit: 100 quota_over: -89 modify_ts: 2025-08-18.15:44:29 $ #
Disable the quota set on the "public" group. This causes $ # iRODS to
hide the quota information we just calculated. $ # This DOES NOT erase
it from the catalog. $ iadmin sgq public total 0 $ iadmin lq Per
resource quotas (if any) for users/groups: No rows found Global (total
usage) quotas (if any) for users/groups: No rows found $ # Re-enable the
quota for the "public" group. Notice that $ # the information is still
there in the catalog from the $ # last run of "iadmin cu". $ iadmin sgq
public total 100 $ iadmin lq Per resource quotas (if any) for
users/groups: No rows found Global (total usage) quotas (if any) for
users/groups: user_name: public zone_name: tempZone quota_limit: 100
quota_over: -89 modify_ts: 2025-08-18.15:45:06 ``` We need to document
this behavior in the help text of `iadmin` and/or add a section to
docs.irods.org. Is this confusing to admins? Should disabling or re-
enabling a group quota clear the original values? This feels like an
optimization, which is fine as long as it's documented. However, if
admins find this surprising, then we need to consider whether the
behavior should be changed.
- [Server silently ignores negative integers passed as the replica number when opening a replica](https://github.com/irods/irods/issues/8617)
**State:** open
**Labels:** bug
**Description:**
github.com/irods/irods_client_http_api/pull/454#discussion_r2280126746
for more information. This was observed while writing tests for the
HTTP API. The iRODS server version was 5.0.1. The iRODS server needs to
report an error when the user passes a negative integer as the replica
number.
#### Closed Issues – irods/irods
- Closed on **2025-09-03**:
[iCommands cannot handle the exclamation mark in logical paths](https://github.com/irods/irods/issues/8656)
- Closed on **2025-09-08**:
[Remove unused `packaging/generate_checksums.sh`](https://github.com/irods/irods/issues/8635)
- Closed on **2025-08-29**:
[`iadmin get_grid_configuration`: Document `access_time` grid configuration options](https://github.com/irods/irods/issues/8630)
- Closed on **2025-08-22**:
[irsync -a does not have tests](https://github.com/irods/irods/issues/8613)
- Closed on **2025-08-18**:
[Switch Debian 13 docker image bases from `debian:trixie` to `debian:13`](https://github.com/irods/irods/issues/8611)
- Closed on **2025-08-19**:
[Support Debian 13](https://github.com/irods/irods/issues/8596)
- Closed on **2025-08-19**:
[Bump version numbers for main and 5-0-stable branches](https://github.com/irods/irods/issues/8595)
- Closed on **2025-08-26**:
[Help text for `igroupadmin` presents incorrect parameter listing](https://github.com/irods/irods/issues/8583)
- Closed on **2025-08-29**:
[irodsServer -h does not mention SIGHUP](https://github.com/irods/irods/issues/8582)
- Closed on **2025-08-18**:
[CMake variable for jsoncons library not defined?](https://github.com/irods/irods/issues/8581)
- Closed on **2025-08-29**:
[unattended setup always prompts for HOST in 5.0.1](https://github.com/irods/irods/issues/8577)
- Closed on **2025-09-08**:
[Data objects without any permissions on them aren't visible to GenQuery2](https://github.com/irods/irods/issues/8546)
- Closed on **2025-08-28**:
[deprecate the control plane and irods-grid](https://github.com/irods/irods/issues/8479)
- Closed on **2025-08-28**:
[`irods-grid pause` seems not working as advertised](https://github.com/irods/irods/issues/8477)
- Closed on **2025-08-28**:
[`irods-grid status` command throws `SYS_PIPE_ERROR` error in logs](https://github.com/irods/irods/issues/8470)
- Closed on **2025-08-18**:
[Define formatter for `SQLCHAR[]`](https://github.com/irods/irods/issues/8455)
- Closed on **2025-08-18**:
[Support fmtlib 11+](https://github.com/irods/irods/issues/8454)
- Closed on **2025-08-28**:
[Deprecate convenience functions for dns/hostname cache config properties](https://github.com/irods/irods/issues/8417)
- Closed on **2025-07-31**:
[irsync errors on checksum mismatch instead of overwriting the target](https://github.com/irods/irods/issues/8384)
- Closed on **2025-08-21**:
[irsync -a fails with error message if used with single data object destination](https://github.com/irods/irods/issues/8295)
- Closed on **2025-08-18**:
[Support EL10](https://github.com/irods/irods/issues/8099)
- Closed on **2025-08-22**:
[GenQuery2: Allow use of nested/multi-argument functions in GROUP BY clause](https://github.com/irods/irods/issues/8093)
- Closed on **2025-08-28**:
[Simplify error message if iRODS env file not readable](https://github.com/irods/irods/issues/8083)
- Closed on **2025-07-31**:
[irsync creates target collection when using -l option](https://github.com/irods/irods/issues/7774)
- Closed on **2025-08-28**:
[How does `irods-grid` interact with tree voting and resources marked down](https://github.com/irods/irods/issues/6750)
- Closed on **2025-08-28**:
[irods-grid is sensitive to ordering of entries in /etc/hosts](https://github.com/irods/irods/issues/6721)
- Closed on **2025-08-28**:
[Main server process does not honor log level changes following `irods-grid --all reload`](https://github.com/irods/irods/issues/6608)
- Closed on **2025-08-28**:
[irods-grid: cannot find libboost_iostreams](https://github.com/irods/irods/issues/6052)
- Closed on **2025-08-28**:
[irods-grid capability to refresh hierarchies of all irodsAgents](https://github.com/irods/irods/issues/3794)
- Closed on **2025-08-28**:
[allow irods-grid command to query status on a paused server](https://github.com/irods/irods/issues/3599)
### Python iRODS Client (`irods/python-irodsclient`)
#### Open Issues – irods/python-irodsclient
- [client tracks current AVU list too zealously](https://github.com/irods/python-irodsclient/issues/768)
**State:** open
**Labels:** enhancement
**Description:**
_reset_metadata to set the _meta attribute to the current list of AVUs.
This is potentially inefficient for large AVU lists. We probably can make
the `_meta` object refresh lazily instead.
- [test that configuration may be influenced via the designed-in environment variables.](https://github.com/irods/python-irodsclient/issues/767)
**State:** open
**Labels:** testing
**Description:**
PYTHON_IRODSCLIENT_CONFIG__GENQUERY1__IRODS_QUERY_LIMIT=128 $ python -ic
'import irods.query; print (irods.query.IRODS_QUERY_LIMIT ==
irods.client_configuration.genquery1.irods_query_limit == 128)' ```
produces an output of `True`. Only, it might be advisable to make the tests
even more general than that.
- [Raise fatal exception for configuration errors.](https://github.com/irods/python-irodsclient/issues/766)
**State:** open
**Labels:** enhancement
**Description:**
should abort on loading (ie when `import irods` executes.) As of now, even
if the loading of a configuration setting causes errors, they are masked and
rendered nonfatal [here](https://github.com/irods/python-irodsclient/blob/80
0fd95a24f153866b64deac18add01578e182fe/irods/client_configuration/__init__.p
y#L302).
#### Closed Issues – irods/python-irodsclient
- Closed on **2025-08-27**:
[Python3 apt package ubuntu 24](https://github.com/irods/python-irodsclient/issues/765)
- Closed on **2025-08-27**:
[cannot make local zone user with same name as an existing remote user](https://github.com/irods/python-irodsclient/issues/764)
- Closed on **2025-08-27**:
[user_zone is ignored by create_user_with_password](https://github.com/irods/python-irodsclient/issues/759)
- Closed on **2025-08-27**:
[Add mypy to static analyzers](https://github.com/irods/python-irodsclient/issues/744)
- Closed on **2025-08-27**:
[Bump iRODS compatibility version](https://github.com/irods/python-irodsclient/issues/743)
- Closed on **2025-08-26**:
[iRODS 5: Update PRC to use new `group` keyword for `rcGeneralAdmin`](https://github.com/irods/python-irodsclient/issues/742)
- Closed on **2025-08-27**:
[Allow default query limit to be set through env variable instead of being hardcoded to 500](https://github.com/irods/python-irodsclient/issues/712)
- Closed on **2025-08-28**:
[release activities for 3.2.0](https://github.com/irods/python-irodsclient/issues/696)
- Closed on **2025-08-21**:
[Add support for pam_interactive auth plugin](https://github.com/irods/python-irodsclient/issues/653)
- Closed on **2025-08-25**:
[Removing metadata](https://github.com/irods/python-irodsclient/issues/157)
- Closed on **2025-08-24**:
[add large file retrieval example to README](https://github.com/irods/python-irodsclient/issues/146)
### NFSRODS (`irods/irods_client_nfsrods`)
#### Open Issues – irods/irods_client_nfsrods
#### Closed Issues – irods/irods_client_nfsrods
### icommands (`irods/irods_client_icommands`)
#### Open Issues – irods/irods_client_icommands
#### Closed Issues – irods/irods_client_icommands
### externals (`irods/externals`)
#### Open Issues – irods/externals
#### Closed Issues – irods/externals
### YODA (`UtrechtUniversity/yoda`)
#### Open Issues – UtrechtUniversity/yoda
- [Base DOI should be generated with first version of dataset](https://github.com/UtrechtUniversity/yoda/issues/605)
**State:** open
**Labels:** No Labels
**Description:** ### Is there an existing feature request for this?
- [x] I have searched the existing feature requests
### Is your feature request related to a problem? Please describe.
Currently, a data publication is initially assigned a single DOI. When a new version is published, that version receives its own DOI, and a new 'base DOI' is also created to track all versions. This base DOI always points to the latest version of the data package.
### Describe the solution you'd like
It might be useful if the 'base DOI' were created at the same time as the first version of the publication, rather than being introduced only when a second version is released. This would allow for consistent citation practices and make it easier to track the evolution of the dataset over time.
It is useful for researchers to use a single DOI in their publications, as it simplifies citation, ensures consistency across references, and reduces confusion when datasets are updated or versioned. By pointing to a 'base DOI' that always resolves to the latest version, researchers can confidently cite data without worrying about version mismatches, while still allowing others to access previous versions if needed.
### Describe alternatives you've considered
Both the base DOI and the version DOI are visible immediately when the dataset is published, allowing researchers to choose which one to use in their scientific publication.
### Additional Context
_No response_
### Acceptance Criteria
Both the base DOI and the version DOI are visible immediately when the dataset is published, allowing researchers to choose which one to use in their scientific publication.
#### Closed Issues – UtrechtUniversity/yoda
- Closed on **2025-09-02**:
[add another organisation to the Affiliation name* in the default-3 metadata form](https://github.com/UtrechtUniversity/yoda/issues/604)
---
If you think someone else would appreciate this newsletter, they can sign up at [https://theresource.metadata.school/](https://theresource.metadata.school/)
One Yak was shaved in the making of this newsletter. Unfortunately a herd approaches over the horizon.