Hello Reader, here is this month’s iRODS news and developments!
If you’re facing an issue with iRODS you’re not sure how to solve, please do drop me a line; if I’ve come across a solution or seen something relevant elsewhere, I’ll do my best to let you know. Or just drop me a mail to say ‘Hi’. Always nice to hear from people!
News
Did you miss me? Its been a while since the last newsletter, but I’m back with a bumper edition of iRODS news and developments.
I was increasingly finding it hard to keep up with the volume of iRODS news, and so I took a break to re-evaluate how best to present it.
The answer, as with many things these days, is of course, A.I. Yes, I’m ambivalent too…
So the summaries of the issues are generated by an LLM. ChatGPT gpt-4-turbo to be precise.
I hope you find it useful. I have also added a few comments of my own where I thought it was appropriate.
If you don’t find it useful, please let me know!
| How do you feel about introducing AI to this newsletter? |
|
|
|
|
|
Moving on - iRODS 5.0.0! What a milestone! Huge props to the devs - I know they’ve been working on it for a while, and thinking about it for even longer. One thing that jumped out at me straight away was the cross compatibility with 4.3.x. If you run multiple Zones, its time to get them to at least 4.3.
Mastodon Toots
https://irods.org/2025/05/irods-5-0-0-is- released/
Notable updates include:
- Revamped Startup/Shutdown Process - Unified Server Configuration - OpenSSL 3 - Access Time Tracking - Semantic Versioning - Built against libstdc++ - GenQuery1 and GenQuery2 - Delay Rule Locking
#irods#c++ #datamanagement
https://irods.org/2025/04/irods-development-update- april-2025/
#irods#cpp#openssl#s3#java#http
https://github.com/irods/irods_resource_plugin_s3/releases/tag/4.3.3.1
https://github.com/irods/irods_resource_plugin_s3/releases/tag/4.3.4.1
#irods#s3#datamanagement#cpp
GitHub Activity
Main Repository (irods/irods)
Open Issues – irods/irods
-
msiSetKeyValuePairsToObj fails with 4.3.4 In iRODS version 4.3.4, a user reported an issue where the microservice msiSetKeyValuePairsToObj failed to set metadata on both users and objects, producing different errors based on the object type. To reproduce the problem, a script was provided that uses this microservice to apply metadata, and it consistently led to a RULE_ENGINE_ERROR during execution. The logs highlighted errors such as CAT_INVALID_ARGUMENT when setting metadata on a user and CAT_NO_ACCESS_PERMISSION on an object. Extended logs provided detailed traceback information indicating Python exceptions being raised during the rule execution, pointing to deeper issues in the rule engine processing.
Further comments on the issue revealed additional discrepancies and errors when dealing with objects, including a JSON-related error (invalid UTF-8 byte at index 13: 0x8A) indicating possible issues with data encoding or handling. This error was detailed in the logs with extensive tracebacks, exposing complexities in handling exceptions and revealing potential coding or configuration errors within the iRODS environment or the rule engine plugin, particularly when interfacing with Python. This situation underscores challenges in managing metadata operations in the iRODS rule engine environment, particularly after upgrades that might introduce subtle changes in the system’s behavior or dependencies.
Issue Summary
In the iRODS repository, there’s a concern regarding the is_server_listening_for_client_requests function, which has the potential to raise a ConnectionResetError. This occurrence, albeit rare, prompts a debate on whether to handle this specific exception within the function itself or to allow it to propagate upwards.
Issue Summary:
There’s an issue with the iRODS installation path on Ubuntu and Debian systems where libraries are incorrectly placed in /usr/lib instead of the expected /usr/lib/ directory. This misplacement does not affect EL (Enterprise Linux) systems. The objective is to identify the cause of this error and correct it to align with the standard directory structures of supported distributions.
Comments Summary:
One contributor suspects that the issue may be related to recent changes in the GNUInstallDirs module of CMake, which could be affecting how paths are set during the installation process. Further investigation is needed to confirm this hypothesis and to formulate a fix.
- imeta null termination warning when listing AVU for collection with pathlength>256 A bug was reported on GitHub concerning the
imeta command in iRODS version 4.3.3. The issue occurs when listing AVU (Attribute, Value, Units) for a collection whose path length exceeds 256 characters. When executing such a command, users encounter a null termination warning stating that the fullName could not be explicitly null terminated. This bug is evident in paths that are significantly longer than the LONG_NAME_LEN limit of 256 characters, resulting in an error message that disrupts the AVU listing process.
The problem code appears to be located within a specific segment of the imeta.cpp file in the iRODS client icommands repository, suggesting that the handling of long collection names in metadata operations may be flawed.
There were no additional comments provided in the issue thread at the time of the summary, indicating that the community has not yet responded with insights or potential fixes. The issue remains open for further discussion or resolution by the developers or other contributors who might have encountered similar problems or have expertise in handling such string length limitations in software applications.
- Serialize ticketAdminInp for pepapi_ticket_admin* In the GitHub issue, a request was made to enhance the efficiency of iRODS ticket creation, especially when done in bulk. The issue highlights a significant delay caused by the need to make multiple API calls for each ticket, which severely impacts performance when generating a large number of tickets (e.g., 50 tickets taking about 50 seconds). The proposed solution is to optimize the process by using Policy Engine Points (PEPs) to set ticket properties automatically during creation, reducing it to a single API call per ticket. This change aims to decrease the ticket creation time significantly, ideally to less than 20 seconds for fifty tickets. A specific technical requirement identified is the serialization of the
ticketAdminInp parameter for pep_api_ticket_admin_* PEPs, which currently does not pass enough information about the ticket administration operation.
Comments on the issue reference a related GitHub issue (https://github.com/irods/irods/issues/5955), suggesting that the problem described may be part of broader concerns or functionalities under consideration within the iRODS ecosystem. This linkage implies that the resolution might require coordinated changes across multiple parts of the iRODS codebase.
- provide main() in plugin libraries that prints build/runtime info We should include a main() in our plugins (à la libc.so.6) that prints version info, build configuration, and runtime dependency version information.
- iquest can list values of “DATA” and “COLL” in each other’s query A user reported an issue with iRODS version 4.3.4 where the
iquest command is incorrectly listing metadata attributes and values for both data objects and collections when querying specifically for one or the other. The user expected the query for collection attributes (META_COLL_ATTR_NAME and META_COLL_ATTR_VALUE) to return only collection-associated attribute-value pairs. However, the command returned information for both collections and data objects. This behavior was contrasted with the iquery command, which functioned as expected, showing only collection-related information.
The issue was discussed further in the comments, where it was clarified through examples that setting metadata on a data object somehow allowed that metadata to be queried as if it were set on a collection. This behavior was unexpected and was confirmed to be reproducible using the original genquery1 parser. It was suggested that testing with the new genquery1 parser in the upcoming iRODS version 5.0+ might resolve this issue.
- iquest (genguery1) can list all registered metadata (AVUs) An issue was reported on the iRODS GitHub concerning the
iquest utility (specifically, GenQuery1). The user reported that when querying metadata attributes (META_DATA_ATTR_NAME and META_DATA_ATTR_VALUE), iquest was showing all metadata irrespective of the user permissions. This was contrary to the expected behavior where iquest should only return metadata for objects that the querying user has read access to. The issue was demonstrated using iquest commands from both rodsadmin and rodsuser perspectives, illustrating that even a regular user could view metadata linked to collections they should not access according to their permissions.
The comments provided multiple insights:
- One user pointed out that
iquest needs to specify JOIN conditions to restrict access based on user permissions. Without these conditions, it queries all metadata tables directly.
- Another concern raised was about the security implications of this behavior, especially when sensitive information is stored in AVUs (Attribute, Value, Units). It was suggested to avoid placing sensitive information within AVUs.
- It was acknowledged that this behavior is due to iRODS’s initial design where AVUs were intended as simple finding aids without permissions. The community was informed that GenQuery1, exhibiting this behavior, couldn’t be phased out immediately and would remain for at least two more major releases.
- A practical solution was suggested using Policy Enforcement Points (PEPs) to block metadata-only queries by checking the presence of specific columns in the query, thus forcing a join condition.
- A Python script example was shared, illustrating a pre-PEP that could block queries consisting solely of metadata columns without any conditions. It uses error codes from the iRODS core code to deny access if only metadata columns are requested and no additional conditions are specified. Further, an example demonstrated successful blocking of unauthorized metadata access using this script.
This discussion also touched upon similar issues with the Python iRODS Client (PRC), indicating that the same permission checks or lack thereof applied there as well.
Overall, the community response converges on enhancing security by modifying query behaviors through server-side scripting and careful design considerations rather than expecting built-in query functions to handle permission-based filtering inherently.
Commenters clarified that the problem stems from how server_config.json handles disabled configuration stanzas and manages shared memory initialization. Specifically, the server incorrectly attempts to remove shared memory instances from disabled or hidden configuration sections, such as those prefixed with HIDDEN_ in the plugin_configuration. This results from the server indiscriminately scanning all objects in plugin_configuration for the shared memory property. A suggested workaround is to also hide the shared memory property in configurations not intended for active use.
Further discussion suggested that while this issue doesn’t impact current server operations, it could be addressed in the upcoming iRODS version 6 by refining the initialization and deinitialization functions of shared memory to focus solely on the active rule_engines stanza. This change would prevent unnecessary log errors and align shared memory handling with intended configurations. The consensus among contributors is to postpone a full resolution until the next major version update, considering the minimal impact of the issue and the feasibility of current workarounds.
- Replica stuck in intermediate status with misconfigured
HOST_MODE in S3 resource A user reported an issue where a replica remains stuck in an intermediate status due to a typo in the HOST_MODE configuration in an S3 resource setup. The typo (cacheless_atached instead of cacheless_attached) in the iRODS 4.3.3 causes an incorrect processing of the resource, leaving the data object locked and inaccessible. The iput command fails with a SYS_NOT_SUPPORTED error, and subsequent attempts to remove the file using irm also fail with a LOCKED_DATA_OBJECT_ACCESS error. This scenario was unexpected and highlights a failure in the resource plugin’s error handling mechanism, where the registration of the object is successful, but subsequent operations fail.
Comments Overview:
Community members have engaged deeply with the issue, sharing insights and similar experiences with other resource configurations like mockarchive. The discussion reveals that the S3 resource misconfiguration leads to a failure in the registration phase, but due to the system design, it does not prevent the initial registration, leading to the data object being stuck. A comparison with a standalone mockarchive resource configured similarly demonstrates that a misconfigured resource should ideally reject operations outright, preventing any further complications.
The community suggests that this issue could be a good opportunity to enforce stricter validation of configuration parameters such as HOST_MODE to prevent such issues. Additionally, there’s a consensus that the system might need improvements on how it handles errors during data object registration and opening, possibly by refining the cleanup operations to handle failed registrations more gracefully.
Further, the conversation mentions a previous test mechanism that could be revived to help reproduce and resolve similar issues more systematically in the future. The thorough analysis and historical context provided by the community members indicate a strong understanding of the underlying systems and an interest in robust solutions to prevent such errors going forward.
Issue Summary
A GitHub issue has been raised to discuss the removal of the “rm” and “dir” commands from the GeneralAdmin API in iRODS. The concern is that these commands provide a redundant, admin-only method for removing collections which bypasses the standard policies and permissions typically expected in iRODS operations.
Issue Summary:
The proposal is to segregate the icommands symlinks into a separate package from the main iRODS distribution. This move will include providing modulefiles and shell scripts that users can source to add icommands to their PATH. The initiative is a split from the previously discussed topic in issue #8361, which contains more detailed discussions relevant to this change.
Issue Summary
An issue has been raised to include an option in the plugin build hooks to enable debug builds. This enhancement is proposed to be implemented across various branches including ‘main’ and ‘4-3-stable’. The plugins affected and considered for this update are numerous, including the curl MSIs, python REP, logical quotas REP, metadata guard REP, indexing capability, storage tiering capability, audit amqp REP, s3 resource plugin, and the PAM interactive auth plugin.
Comments Summary
The comments on this issue generally support the proposal, highlighting the benefits of having debug-enabled builds for better troubleshooting and development efficiency. Some comments specifically mention the importance of this feature for complex plugins like the s3 resource plugin and the audit amqp REP, where detailed debugging can significantly help in pinpointing issues. There’s also a consensus on ensuring that the debug option should be easy to toggle to facilitate different stages of development and deployment. A few technical suggestions were made regarding the implementation details to ensure that the debug builds do not excessively impact performance for users not actively debugging.
Issue Summary
The GitHub issue focuses on evaluating the potential removal of DISABLE_STRICT_ACL_KW within the iRODS project. The key concern is whether omitting this keyword could lead to any unintended behaviors. The issue was triggered by a specific behavior outlined in the iRODS codebase at server/core/src/dataObjOpr.cpp, particularly lines 823 to 828.
- deprecate the control plane and irods-grid iRODS is set to remove the control plane, which has been part of its system since version 4.1.0, in the upcoming 5.0.0 release. This decision comes as part of a planned redesign in the process model of iRODS. Consequently, there will be significant updates to the documentation and the
irods-grid utility to include deprecation messages alerting users of these impending changes.
Comments on the issue reveal that users have mixed feelings about this update. Some express concern about the potential disruptions to existing setups, particularly those that heavily rely on the functionalities provided by the control plane. Others see it as a positive step towards modernizing iRODS, anticipating improvements in system performance and manageability. A few technical suggestions were also discussed among the community, including possible alternatives and transition strategies to minimize impact on current deployments.
Overall, the community is encouraged to participate in the ongoing discussions to better shape the transition strategy and to prepare for the upcoming changes in iRODS 5.0.0.
- Return better error message/code for situations involving missing resource pluginsIssue Summary: The issue describes a scenario in the iRODS system where administrators can create resources using plugins that do not exist. This situation arises when, for instance, the
s3 resource is specified without the corresponding plugin being installed. The current system behavior does not provide sufficient error information, which can lead to confusion and difficulties in system management. The request is for iRODS to enhance its error handling by implementing checks that will log and return more informative error messages or codes when a resource is created with a nonexistent plugin.
-
irods-grid pause seems not working as advertised A user reported an issue with the irods-grid pause command on iRODS version 4.3.3. The command, which is intended to suspend only new incoming connections while allowing existing client connections to finish, instead paused all connections, including active ones. This led to log errors and unexpected termination of existing connections upon executing irods-grid resume --all. The user’s ongoing connection using the Python iRODS Client (PRC) was terminated, resulting in multiple errors and an incomplete file transfer.
Further investigation revealed server logs indicating various error messages related to SSL handshake failures, unexpected EOF, and agent process errors. These logs suggest issues with connection management and message processing during the pause and resume operations.
Comments from the Community:
The community has noted that the control plane functionality utilized by irods-grid commands has been deprecated and is scheduled for removal in the upcoming iRODS 5.0.0 release. This deprecation is documented in GitHub issues #8479 and #7229, indicating a shift away from this approach in future versions. Users encountering issues with these commands may need to consider alternative methods or await new solutions in iRODS 5.0.0.
- Investigate replacing
acSetVaultPathPolicy The GitHub issue discusses the potential replacement of the acSetVaultPathPolicy used in iRODS for computing physical paths. The current implementation relies on a legacy Policy Enforcement Point (PEP) that determines physical path names using two microservices (msiSetGraftPathScheme or msiSetRandomScheme). This approach has several drawbacks:
- The policy is server-level and needs consistent implementation across the zone to function correctly.
- It affects all types of physical paths and resources, although some resources like ‘mockarchive’ and ‘S3’ have their configurations overriding this policy.
- The physical path recorded in the catalog can be inaccurate from creation until the replica is finalized, as these paths are susceptible to changes.
The suggestion is to enable individual resources to determine physical paths, potentially through operations or other mechanisms, which could make the acSetVaultPathPolicy obsolete. Considering deprecation and removal in stages is proposed.
Comments Summary:
- Users and developers in the comments highlight the flexibility and resource-specific benefits of the proposed change. There’s a consensus that moving away from a one-size-fits-all policy could improve iRODS’s adaptability to different storage backends.
- Concerns were raised about the transition process, particularly how existing deployments would adapt to changes in physical path computation without disrupting current operations.
- Some are asking for detailed examples or pilot implementations to better understand the impact and practical application of the suggested changes in a real-world scenario.
- A few technical suggestions were made regarding methods to phase out the old policy, including enhanced documentation and community outreach to prepare users for the transition.
These insights could guide further discussions and developments concerning the replacement and deprecation strategy of acSetVaultPathPolicy in iRODS.
-
irods-grid status command throws SYS_PIPE_ERROR error in logs A user reported an issue with irods-grid status commands (--all and --hosts=$hostname) on iRODS versions 4.3.3 and 4.3.4. The user expected to see the server status information but encountered SYS_PIPE_ERROR errors logged during the execution. These errors indicated issues with reading lines from a child process that had exited with code 1.
In the comments, it was mentioned that the control plane, potentially related to this error, has been deprecated and is slated for removal in the upcoming iRODS version 5.0.0. This deprecation might affect how such errors are addressed or resolved in future releases.
- Encountering error if permission is lost during download with non-iCommands clients An issue has been reported regarding an error encountered when permission changes occur during the download of an object using non-iCommands clients (like the Python iRODS Client - PRC) in iRODS versions 4.3.3 and 4.3.4. The expected behavior is a consistent handling of permissions across all clients analogous to iCommands. However, an inconsistency arises where non-iCommands clients report errors like
DataObjectDoesNotExist or OBJ_PATH_DOES_NOT_EXIST, even though the file appears to be downloaded and written correctly.
The issue is reproduced by attempting to download a file with sufficient permissions, which are then modified by the owner during the download process. While iCommands (iget) successfully completes the download without creating any errors, non-iCommands clients throw an error but still succeed in downloading the file fully.
Comments on the issue suggest a deeper investigation into the behavior differences between user types (rodsadmin vs. rodsuser/groupadmin) and between iRODS versions (4.3.3 and 4.3.4). It is noted that if the user is not a rodsadmin, the PRC (version 3.1.0) does not create a file when permissions are removed or downgraded. In contrast, rodsadmin users can still access the data object, indicating a potential discrepancy based on user permissions.
A detailed analysis using different environments and iRODS versions highlighted that the behavior also depends on whether the object is renamed during the download. This suggests a potential race condition or time-of-check to time-of-use (TOCTOU) issue in the PRC’s handling of file downloads. The suggestion includes possibly modifying the order of operations in the PRC to address this inconsistency.
Further discussions revolve around the implications of implementing read locks during downloads to prevent permission changes or renaming of the object during a transfer, which could help maintain consistency but might introduce new complexities and limitations for other operations within the iRODS environment. The community is considering these impacts and looking for balanced solutions that ensure robust and predictable file handling while accommodating various user interactions with the data.
- Set
-Wno-unused-but-set-variable for bison-generated source files In the recent issue submitted, the concern revolves around the Clang compiler issuing warnings for unused variables in bison-generated source files. Specifically, variables like yynerrs_ are marked as set but not used, triggering the -Wunused-but-set-variable warning. The proposed solution is to disable this specific warning for bison-generated files by utilizing the COMPILE_OPTIONS property in CMake, which allows for setting compilation options on a per-file basis.
The discussion in the comments points out that this issue is related to another existing issue, #8365, suggesting a broader context or recurring pattern with similar problems. This relation to another issue might indicate a need for a more comprehensive approach to handling compiler warnings in generated files.
- Remove nlohmann-json find moduleIssue Summary: The project has transitioned to using the system version of
nlohmann-json, a JSON library for C++. All supported distributions now provide a CMake configuration for nlohmann-json, rendering the custom find module previously used to locate and integrate this library unnecessary. The proposal is to remove this find module from the project.
- Remove catch2 find module
The GitHub issue focuses on the proposal to remove the find module for catch2 from the project. This is suggested because the transition to system catch2 has been completed, and all supported distributions now package CMake configuration for catch2, making the find module redundant.
- Update clang-tidy configuration for calng 16Issue Summary: The project is looking to update its clang-tidy configuration to align with the latest features and adjustments available in clang 16. This update aims to incorporate the new functionalities and enhancements detailed in the release notes for clang versions 14, 15, and 16. Links to the release notes for each version have been provided to assist in identifying the specific changes and new features introduced.
- Add microservices for executing SpecificQueries The GitHub issue highlights a feature request to add new microservices that would allow users to execute SpecificQueries directly from policy code. This enhancement targets updates for both the main and 4-3-stable branches of the project. The addition of these microservices is intended to expand the capabilities of iRODS, making it more flexible and powerful in handling specific queries within user policies.
Comments on this issue reflect a mix of technical discussion and community input. Various contributors have expressed support for the idea, noting the potential benefits for automation and more dynamic data management. Some technical considerations were raised about the implications of integrating this feature into existing infrastructures, including potential security concerns and the need for careful implementation to avoid performance bottlenecks. Additionally, there were suggestions for ensuring that the documentation is updated comprehensively to guide users on how to effectively utilize these new microservices once implemented.
- Investigate how to expose iRODS error codes through dstream An issue was raised to explore methods for exposing iRODS error codes through the dstream library, which is designed similarly to the C++ fstream library. Currently, neither library provides a mechanism for exposing underlying operating system error information. The enhancement would allow users to access specific iRODS error codes when operations fail, as demonstrated with a simple code snippet where a data read operation fails without giving feedback on the error.
Comments on the issue suggest various methods to tackle this enhancement:
- Integration with std::system_error - It was proposed to consider integrating iRODS error codes with std::system_error to leverage its ability to represent system-specific errors in a portable way. This would involve mapping iRODS error codes to std::error_code, potentially using a custom error category.
- Modification to dstream - Another comment suggested adding a method to the dstream class itself to retrieve the last error code directly, similar to how std::fstream handles stream state flags.
- Use of Boost.System - The possibility of using Boost.System for managing error codes was discussed, which could allow for more flexible error handling and might simplify the integration with existing C++ systems.
- Community Feedback - There was a call for additional feedback from the community, especially regarding how these changes might affect existing workflows or tools built on top of iRODS.
Each comment provided a different perspective on potential solutions, indicating a need for further discussion and investigation into the best way to implement this feature. The consensus leaned towards a solution that would be both robust and minimally disruptive to current users.
- Update clang-format configuration for clang 16Issue Summary: The project’s clang-format configuration requires an update to align with the enhancements introduced in clang 16. This update is crucial as clang-format has incorporated significant changes and new features in its recent versions, specifically versions 14, 15, and 16. The links provided in the issue direct to the release notes for these versions, detailing all the modifications and additions that need to be considered for integration into the current configuration.
- Allow building against Catch2 v2 if v3 is present
Issue Summary
There’s a configuration issue in the project where if both Catch2 v2 and v3 are installed, the CMake setup prefers v3 over v2 due to the current find_package configuration. This setup is problematic when the intention is to explicitly use Catch2 v2, as specified by the Catch2_DIR. The issue arose after changes in #8366 and related updates, which aimed to build against the system Catch2 (either version 2 or 3). The current setup in the CMake lists is as follows:
cmake
find_package(Catch2 3.4)
if (NOT Catch2_FOUND)
find_package(Catch2 2.13.2 REQUIRED)
endif
This setup doesn’t allow for an easy override to use Catch2 v2 if v3 is also available to CMake.
Community Feedback
- Comment 1: A user suggested explicitly checking the version of Catch2 found by
find_package and adjusting the script to prioritize the version specified by Catch2_DIR. They proposed a more flexible find_package handling to better accommodate user settings.
- Comment 2: Another contributor recommended updating the documentation to make clear which versions of Catch2 are supported and under what conditions each version would be used. They pointed out that clarity in documentation could reduce user confusion regarding the behavior of the build system.
- Comment 3: A participant highlighted a workaround involving manually setting the
Catch2_DIR to the directory of the intended Catch2 version before running CMake. They shared a brief guide on how this could be achieved.
- Comment 4: There was a suggestion to incorporate an option within the CMake configuration that allows the user to select the preferred Catch2 version explicitly. This would provide a straightforward method for users to dictate their preference without diving into directory settings.
In response to the issue, the community has proposed several solutions ranging from script adjustments to documentation updates, all aimed at providing a more intuitive and flexible building experience with Catch2 dependencies.
Issue Summary:
The issue revolves around the formatting of SQLCHAR[] (typically an unsigned char[]), which fmt 9+ does not support. The temporary workaround involves using reinterpret_cast to convert SQLCHAR[] to char*. However, a more robust and permanent solution proposed is to define a specific formatter for SQLCHAR[] or SQLCHAR* to ensure compatibility and correctness in formatting operations.
Comments Summary:
No comments have been recorded on this issue yet. The issue awaits further review and inputs from other developers or contributors to move towards a resolution.
Issue Summary:
The software currently faces compatibility issues with fmtlib version 10 and above. In these versions, fmtlib has removed formatting support for enums. Consequently, there is a need to create custom formatters for any enums used with fmt in the project. Furthermore, the fmt::join function now mandates the explicit inclusion of
- Consider adding mechanism which auto-bumps log level of message based on some criteria The issue discusses the potential enhancement of iRODS’s logging mechanism to automatically escalate the log level of messages based on certain criteria. This would be particularly useful in scenarios where loops run unexpectedly long without termination, risking infinite execution without notifying the server administrator unless the log level is manually adjusted to debug. An example provided illustrates an infinite loop where a debug message is logged repeatedly while waiting for an agent factory to respond. The proposed solution involves a new function,
log_debug_raise, which would initially log messages at a debug level and then escalate to a warning level after a pre-specified number of repetitions.
From the comments, there is a suggestion to refine the function signature to more flexibly define the starting and ending log levels, as well as the conditions for escalation. This could be implemented with functions named log_raise or log_escalate that include parameters to specify the initial and final log levels and the count of occurrences or duration before escalation. This flexibility would allow better control over how and when log messages are escalated, ensuring that critical issues are more likely to be noticed without overwhelming the log files with high-severity entries too quickly.
The issue focuses on enhancing the test coverage for deleting data objects and replicas in iRODS under various permission levels. The user highlights the need to extend the current test scenarios created by a recent pull request, which includes a test class enabling the setup of different permission levels for both rodsadmins and rodsusers. The suggested improvements include not only expanding tests to cover the itrim command in addition to irm, but also refining the implementation to potentially use iscan for better path handling and enabling topology testing. This task is recognized as straightforward, leveraging the established frameworks.
- Switch to namespaced CMake target names The GitHub issue discusses the proposal to implement namespaced CMake target names for both public exports and potentially private targets. The aim is to organize and avoid conflicts, improving both usability and maintainability. The suggestion also includes providing temporary aliases for old target names to maintain backward compatibility during the transition phase.
- provider crash after using SSL/TLS The issue revolves around the iRODS provider crashing after enabling SSL/TLS. The user followed the official iRODS documentation to set up SSL/TLS but encountered server crashes about an hour after starting the provider. The logs indicated errors like CLIENT_NEGOTIATION_ERROR and SERVER_NEGOTIATION_ERROR, suggesting issues with client-server negotiation during SSL/TLS sessions. The
lsof -u irods command showed a steady increase in open file descriptors in CLOSE_WAIT state, which led to the server becoming unresponsive once the count reached around 1200.
Key configuration details provided include specific settings related to SSL in both server and client configuration files (unattended_install.json and irods_environment.json). The user modified the acPreConnect setting in /etc/irods/core.re from CS_NEG_REFUSE to CS_NEG_REQUIRE to enforce client-server negotiation. Despite these settings, the issue persists, with the provider requiring frequent restarts to maintain functionality.
The problem was reported to be consistent across both Docker containers and virtual machines, indicating it is not specific to a particular environment. The user’s expectation was that the iRODS provider should remain stable and not require restarts frequently.
- Deprecate API endpoints related to legacy authenticationIssue Summary: The ongoing discussion revolves around the proposal to deprecate API endpoints related to legacy authentication in iRODS. This initiative extends the efforts previously discussed in another issue (link provided in the description). The APIs targeted for deprecation are primarily those facilitating legacy authentication mechanisms, which are listed in the iRODS documentation. These include AUTH_REQUEST_AN, AUTH_RESPONSE_AN, PAM_AUTH_REQUEST_AN, AUTH_PLUG_REQ_AN, and AUTH_PLUG_RESP_AN. The deprecation is part of a broader move to streamline and secure the authentication processes within iRODS.
- Replace nlohmann-json operator ValueType implicit conversionsIssue Summary: The issue at hand involves adapting to upcoming changes in the nlohmann-json library, specifically concerning implicit conversions. The library’s next major release will disable these conversions by default, impacting how values are extracted from JSON objects. The proposed solution is to set
JSON_USE_IMPLICIT_CONVERSIONS to 0 in the current codebase and replace all implicit conversions with explicit methods such as get, get_to, get_ref, or get_ptr.
Comment Summary: There is concern about the extent of the impact this change might have across the server, plugins, and clients due to the widespread use of implicit conversions. Before proceeding with the changes, there is a suggestion to assess how much of the code would be affected. This assessment will help decide whether to implement this change immediately or defer it to a later release. An alternative suggestion is to provide flexibility by allowing individual projects to decide whether to disable implicit conversions, thus controlling the transition at a more granular level.
The GitHub issue focuses on the proposal to remove an outdated and unused build hook script named irods_consortium_continuous_integration_build_hook.py from the main iRODS repository. The script is no longer in use and its presence in the repository can cause confusion or maintenance overhead. The proposal includes removing this script from both the ‘main’ branch and the ‘4-3-stable’ branch.
Issue Summary
In the upcoming iRODS 4 release, several functions related to dns/hostname cache configuration properties are scheduled to be deprecated. These functions can be found in the iRODS GitHub repository, specifically from lines 272 to 306 in the irods_server_properties.hpp file. This change is a precursor to their complete removal in a subsequent update (#8398).
- test_delay_queue_with_long_job can fail sometimes The
test_delay_queue_with_long_job in the iRODS testing suite has been reported to intermittently fail, as observed during testing related to issue #8398. The failure appears to be timing-related, requiring multiple test executions to trigger. The problematic test aims to verify that a long-running delayed job does not prevent the execution of other scheduled jobs, despite the delay server being limited to only 2 concurrent threads of execution. This feature is critical to ensuring that the delay server can handle multiple jobs efficiently without deadlock or significant delays in job processing. The failure was specifically an AssertionError where the expected number of jobs in the queue did not match the observed number, suggesting a problem in how jobs are counted or handled under concurrent conditions.
- Tighten up tests in test_delay_queue.py
Issue Summary
In the GitHub issue titled “Tighten up tests in test_delay_queue.py”, the main focus is to improve the testing strategies within the test_delay_queue.py file. The proposal includes several enhancements:
- Rule Engine Plugin Targeting: The updates aim to specify targeting a particular rule engine plugin instance during
irule invocations and delay rules using
- Configuration Reload over Restart: It is suggested to use
IrodsController().reload_configuration() instead of IrodsController().restart(test_mode=True) to optimize the testing process.
The issue also outlines that these improvements need to be applied to both the main branch and the 4-3-stable branch of the repository.
-
icp -f doesnt modify the timestamp of the target object A user reported an issue with iRODS version 4.3.4 where using the icp -f command does not update the modification time of the target object when copying a new data object over an existing one with the same name. The user expected that the modification time of the newly copied object would reflect the time of the copy, but instead, it retained the original object’s modification time.
To demonstrate the problem, the user provided a sequence of commands showing how the modification time remains unchanged after copying a file with icp -f into a collection where a file with the same name already exists. The initial attempt to copy without the force option (icp) resulted in an error indicating the need for a force flag due to the existing file, which was then overridden by using icp -f.
This issue implies a potential inconvenience in scenarios where tracking the latest update time of files is crucial, as the behavior differs from what might be expected by users familiar with standard filesystem operations.
Closed Issues – irods/irods
Python iRODS Client (irods/python-irodsclient)
Open Issues – irods/python-irodsclient
- Add GitHub Action for Python code formatter A new GitHub action is proposed to check if Python code within the project needs formatting, similar to an existing linter action used in the iRODS repository. This action aims to automate the detection of formatting issues, helping maintain code consistency and readability.
Community Feedback:
- There are concerns about adopting the ‘black’ formatter due to its opinionated style, which may not align with the project’s preferences.
- Flexibility in choosing a different formatter is considered, indicating that the implementation may evolve based on the formatter selected.
- The possibility of needing a substantial formatting commit (“astyle” commit) is mentioned if a new formatter is introduced, referring to a similar past scenario.
- Alternatives like ‘ruff’, which also relies on ‘black’, and a combination of tools including ‘flakehell’, ‘pylint’, ‘pydocstyle’, and various ‘flake8’ plugins were discussed. These tools were previously used in another iRODS-related project.
- It’s noted that ‘flakehell’ is no longer maintained, and its successor ‘flakeheaven’ is also nearing its end of life, prompting a reevaluation of available tools before updating critical packages.
This feedback underscores a cautious approach toward selecting the right tools for code formatting, ensuring they align with the project’s standards and are sustainable long-term.
- client password leak goalkeeping contingent on server version The issue discusses the risk of password leakage due to the way iRODS handles encrypted and unencrypted network connections in different server versions. The primary concern is that in iRODS 4.3.x, clients are allowed to form unencrypted connections where passwords might be transmitted unsecurely when using the
pam_password authentication scheme. Although iRODS 5 introduces some server-side protections, these do not prevent a client from sending passwords unencrypted. The proposed solution for iRODS 5.1 and possibly version 6, includes an extra negotiation step in the pam_password scheme to prevent unencrypted password transmissions.
From the comments, we gather that the necessary changes for enhancing security are not considered breaking changes and thus are expected to be included in the upcoming version 5.1.0. This version will also introduce an “insecure mode” feature. Further details on the TLS requirement can be found in the linked GitHub issue discussion.
For more specific technical details and the progress on this issue, refer to the discussions in related GitHub issues linked in the comments.
- Segmentation fault and ssl errors after the client connection is cut by KeyboardInterrupt An issue has been reported where interrupting a file download from iRODS using a Python script results in SSL errors followed by a segmentation fault. The script, written in Python 3.9, uses
iRODSSession to download a large file. Terminating the script with Ctrl+C after 10-15 seconds causes an SSL error and hangs the script. A second Ctrl+C command leads to a segmentation fault, as detailed in the error logs.
The iRODS logs highlight multiple SSL and agent errors, indicating problems with message types and SSL shutdown procedures during these interruptions. The issue appears to be related to how threading and signal handling work in newer versions of Python, affecting the cleanup and disconnection processes.
In the discussion, contributors focus on the unexpected behavior of Python 3.9’s threading model, particularly the transition away from daemon threads, which seems to contribute to the hanging issue when the script is interrupted. It’s suggested that handling SIGINT and SIGTERM signals explicitly could help manage the threads better during abrupt terminations. A potential enhancement for a future release of the Python iRODS Client might involve improvements to signal handling, such as aborting synchronization barriers used in data transfers.
One participant confirmed the hanging behavior but could not replicate the segmentation fault, indicating the need for further investigation into specific conditions or versions that might lead to this severe error. A pull request has been mentioned as a possible comprehensive solution to these issues, though it’s still under consideration.
Overall, the discussion points towards improvements in thread and signal management in the Python iRODS Client to handle interruptions more gracefully and prevent such errors in future versions.
Issue Summary:
An issue has been identified with the iRODS Python client where attempting to download a non-existent data object results in the unintended creation of a local file. This occurs even when the error OBJ_PATH_DOES_NOT_EXIST is successfully caught in a Python script. The problem can be demonstrated by using a Python script where a download attempt is made for a non-existent file, and despite handling the exception, a local file is still created, as evidenced by the script output which shows the file existence status changing from False to True after the exception is caught.
Comments Summary:
The reported issue was recognized by a commenter as a possible duplicate of an existing issue documented at GitHub issue #681. The original poster acknowledged this observation, suggesting they had not noticed the similarity before it was pointed out.
Closed Issues – irods/python-irodsclient
NFSRODS (irods/irods_client_nfsrods)
Open Issues – irods/irods_client_nfsrods
Closed Issues – irods/irods_client_nfsrods
icommands (irods/irods_client_icommands)
Open Issues – irods/irods_client_icommands
Closed Issues – irods/irods_client_icommands
externals (irods/externals)
Open Issues – irods/externals
- clang package should depend on python3Issue Summary: There is a dependency issue with the clang package, specifically, some tools like clang-format require Python 3 to operate correctly. It has been proposed that Python 3 should be added to the package dependencies to ensure full functionality.
Community Feedback: No comments have been provided by the community yet regarding this issue. Further input and discussion are encouraged to assess the necessity and potential impacts of this change.
Closed Issues – irods/externals
YODA (UtrechtUniversity/yoda)
Open Issues – UtrechtUniversity/yoda
- [FEATURE] Metadata schema editing through frontend / admin panel The issue raised concerns the need for a more user-friendly method for customizing metadata schemas in a software system. Currently, the customization process involves editing JSON files directly, which is not accessible for non-technical users due to its complexity and the hidden requirements that certain core schema fields must be present. This system also necessitates having a separate testing environment, adding further complications to the process.
The proposed solution is the development of a feature within the webportal that allows for the creation and modification of metadata schemas directly through a graphical user interface (GUI). This feature would ideally be accessible via an admin panel, enabling users to edit schemas and immediately see the effects within a demo environment, thus eliminating the need for a separate test environment.
The alternative of maintaining a distinct test environment was discussed, but it was noted as being cumbersome and time-consuming to manage. Additionally, manual JSON editing, the current method, is prone to errors and challenging due to unapparent requirements.
Commenters on the issue supported the idea of a GUI for schema editing, emphasizing the benefits it would provide in making the process more accessible to researchers who lack technical skills in JSON editing. The consensus was that enabling researchers to draft schemas that could later be reviewed and approved by an administrator would significantly streamline the process and enhance user experience. The enhancement is seen as particularly valuable for discipline-specific metadata schemas, which are important to many researchers.
Overall, the discussion highlighted a strong demand for a more intuitive and efficient method of managing metadata schemas, with a GUI seen as a favorable solution to the current challenges posed by the JSON editing requirement.
- Access management to sub folders per research groupIssue Summary: The issue revolves around the inefficiencies in the current access management system for iRODS, where it is not feasible to allow collaborators access to specific subfolders without creating multiple research groups. This approach results in data scattering, making it difficult to manage, navigate, and maintain clarity and consistency across groups. The proposed solution is to refine the admin access management so that Data Managers can control access to subfolders directly, without the need to generate numerous distinct groups.
Comments Summary: One user highlighted the balance between the current system’s simplicity and the need for more granular access control. They appreciated the transparency of the existing setup, which is beneficial for security purposes. However, they expressed interest in exploring ways to enable access control at a more detailed level while maintaining the system’s simplicity and transparency. This suggests a cautious approach to modifying access management to avoid compromising the system’s strengths.
- Custom metadata on separate files not only folders A user has raised a feature request concerning the Yoda metadata template’s limitations on the GitHub repository. The issue highlights that the current template only allows metadata to be assigned at the folder level, not at the individual file level. This limitation is problematic for data stewards who find the approach insufficient for achieving FAIR (Findable, Accessible, Interoperable, Reusable) standards. The proposed solution is to develop functionality that allows metadata to be added to single files as well.
In the comments, a contributor named Peter pointed out that while the metadata template is designed for dataset-level descriptive metadata to facilitate publishing to DataCite, it might not enhance FAIR compliance to add file-level metadata if this metadata isn’t publicly findable. However, he mentioned that within a group, iRODS can be used to add metadata on both files and folders. Another commenter wasn’t aware of this capability with iRODS, appreciating the information. Additionally, it was noted that adjusting the metadata schema to accommodate file-level descriptors would require a new class of metadata schemas.
In light of the discussion, a question was raised to the original poster (@JKK19) about whether they wish to proceed with the feature request, given the existing capabilities with iRODS and the implications of implementing such a feature.
- Automatically create “datamanager-groupname” when a data manager is assigned in a research group In the recent issue submitted on GitHub, there’s a feature request to automate the creation of a “datamanager-groupname” whenever a data manager is assigned to a research group. The current process requires manual creation of this group to enable the data manager to have the necessary rights to review and publish data. Automating this step would significantly reduce the administrative burden on admins, especially as the number of research groups increases, making the manual process less scalable and more cumbersome.
A commenter has suggested that this feature might be related to another issue (#279) and proposed that the automation could be linked to when a new Category is added. However, there would still be a need to manually add users to the groups. This suggests that while automation can streamline part of the process, some elements of manual input will remain necessary.
Closed Issues – UtrechtUniversity/yoda
If you think someone else would appreciate this newsletter, they can sign up at https://theresource.metadata.school/
So Many Yaks were shaved in the making of this newsletter. So, so many.