VFBquery v0.5.7

VFBquery v0.5.7 Release Notes

New Features

Imaging Technique Support for Individual Terms

• Added Technique field to term info output for Individual terms (neurons, expression patterns, etc.)
• Automatic technique extraction from VFB_JSON data sources:
  • Channel images (channel_image.imaging_technique)
  • Anatomy channel images (anatomy_channel_image.channel_image.imaging_technique)
  • Relationships with OBI_0000312 (is_specified_output_of) predicate
• Conditional query generation for computer graphic techniques:
  • Individuals with "computer graphic" techniques now automatically trigger parent class queries
  • Enhances anatomical querying capabilities for painted domains and synthetic data

Painted Domain Detection

• Added IsPaintedDomain field to identify terms representing computer-generated anatomical domains
• Automatic detection from imaging technique metadata
• Enhanced query logic for synthetic anatomical structures

Technical Improvements

Schema Updates

• Updated TermInfoOutputSchema to include Technique and IsPaintedDomain fields
• Made Technique field optional in JSON schema (Individual terms only)
• Updated documentation examples and schema definitions

Code Refactoring

• Refactored term_info_parse_object function for better modularity
• Enhanced relationship processing for technique extraction
• Improved query generation logic for anatomical classes

Testing & Validation

• Updated test validation to handle optional Technique field
• Fixed CI/CD pipeline to properly validate term info structure
• Maintained backward compatibility for existing API consumers

Documentation Updates

• Updated README.md with Technique field examples
• Updated schema.md with complete field definitions
• Added examples showing technique information for Individual terms

Performance & Caching

• Continued optimization of SOLR caching layer
• Updated performance test baselines
• Enhanced caching for painted domain queries

Bug Fixes

• Fixed test validation for optional fields in term info output
• Improved error handling in query result processing

Compatibility

• Backward Compatible: All existing API calls continue to work unchanged
• Optional Fields: New fields are optional and won't break existing integrations
• Python 3.8+ support maintained

Migration Notes

• No breaking changes - existing code will continue to work
• New Technique field will appear in Individual term results when imaging techniques are available
• IsPaintedDomain field provides additional metadata for computer-generated anatomy

---

This release enhances VFBquery's ability to provide rich metadata about imaging techniques and synthetic anatomical data, improving the user experience for researchers working with diverse data types in VirtualFlyBrain.

VFBquery v0.5.6

VFBquery v0.5.6 Release Notes

Performance Improvements

• Enhanced Caching System: Added caching for painted domains and warm-up queries in performance tests to improve query execution times
• Increased Performance Thresholds: Updated performance test thresholds to accommodate improved caching and query optimisations
• Query Warm-up Caching: Implemented caching for query warm-up operations in test suites to reduce initialisation overhead
• Additional Query Types Caching: Extended the with_solr_cache function to support caching for more query types

Test Suite Updates

• Updated Test Data: Refreshed test queries to use specific, stable term IDs for:
  • Morphology comparisons
  • Anatomy single-cell RNA sequencing queries
  • Cluster expression queries
  • Gene IDs and dataset terms in performance tests
• Error Handling Improvements: Added robust error handling for None query results in term info serialisation tests
• Logging Enhancements: Implemented logging in query functions to improve debugging and monitoring capabilities

Bug Fixes and Stability

• Query Result Validation: Enhanced validation for term info results to handle edge cases and invalid query structures
• Exception Handling: Improved graceful handling of exceptions in query functions during serialisation processes

Technical Details

• All performance test results have been updated to reflect the latest optimisations
• Maintained backward compatibility with existing API interfaces
• Enhanced test coverage for error conditions and edge cases

This release focuses on performance optimisations and improved reliability of the VFBquery library, particularly for applications requiring high-throughput querying of VirtualFlyBrain data.

VFBquery v0.5.5

VFBquery Release Notes - v0.5.5

Bug Fixes

• Fixed "Unsupported result format" error: Resolved crashes in term info processing when query functions return None instead of expected data structures. Added proper error handling in fill_query_results to gracefully skip invalid results.

• Corrected neuron connectivity counts: Fixed get_neuron_neuron_connectivity to return accurate total counts in preview mode, regardless of limit and direction parameters. Previously, counts were incorrectly showing limited row counts instead of total connected neurons.

• Improved direction filtering: Enhanced the neuron connectivity query to properly filter by connection direction ('upstream', 'downstream', or 'both') in both the main query and count query, ensuring consistent results.

Improvements

• Standardised API defaults: Changed get_neuron_neuron_connectivity default parameter from return_dataframe=False to return_dataframe=True for consistency with other query functions in the library.

• Enhanced count accuracy: Implemented separate count queries for all query functions that return dict format, ensuring preview counts always reflect total available results even when limited.

Technical Details

• Added proper Cypher count queries that respect direction filtering parameters
• Improved error handling for query failures in term info processing
• Maintained backward compatibility - all existing code will continue to work
• All tests pass, including neuron connectivity and term info serialisation tests

Compatibility

• Backwards Compatible: No breaking changes to existing APIs
• Dependencies: No new dependencies required
• Python: Compatible with existing Python 3.x requirements

Contributors

• Thanks to the VFB team for identifying and testing these improvements

---

For more details on specific changes, see the commit history or issue tracker. If you encounter any issues, please report them on the VFBquery GitHub repository.

VFBquery v0.5.4

VFBquery v0.5.4 Release Notes

Bug Fixes

SOLR Caching Result Format Separation

• Fixed: SOLR caching system now properly separates DataFrame and dict results for all query types that support the return_dataframe parameter
  • Added missing query types to the dataframe_query_types list: 'neuron_neuron_connectivity_query', 'neuron_region_connectivity_query', 'instances', 'templates', 'images_neurons', 'images_that_develop_from', 'expression_pattern_fragments'
  • This ensures cached DataFrame results are not returned when dict results are requested, preventing format mismatch errors
• Impact: Completely resolves the "Unsupported result format for filtering columns" errors that were still occurring in some term info queries
• Root Cause: The caching system was using the same cache key for DataFrame and dict versions of query results, causing format confusion when fill_query_results() expected dict results but received cached DataFrames

Performance Updates

• Updated performance test results documentation

Technical Details

• Modified with_solr_cache() decorator in solr_result_cache.py to include return_dataframe parameter in cache keys for all applicable query types
• Cache keys now include _dataframe_{True/False} suffix for queries that support format selection
• This ensures complete isolation between different result formats in the cache

Testing

• Verified that term info queries for individual neurons with connectivity information now work correctly
• Confirmed that caching properly handles both DataFrame and dict result formats
• All existing functionality remains intact

This release provides a complete fix for the query result format issues that were affecting term info preview generation, ensuring reliable operation of the VFBquery library.

VFBquery v0.5.3

VFBquery v0.5.3 Release Notes

Bug Fixes

Query Result Format Consistency

• Fixed: Inconsistent return format in get_neuron_neuron_connectivity() and get_neuron_region_connectivity() functions
  • Changed return dictionary key from 'data' to 'rows' to match other query functions
  • Added backward compatibility handling for any cached results still using the old 'data' key
• Impact: Resolves "Unsupported result format for filtering columns" errors when populating preview results in term info queries
• Affected Functions:
  • get_neuron_neuron_connectivity()
  • get_neuron_region_connectivity()
  • fill_query_results() (enhanced with legacy support)

Performance Updates

• Updated performance test results documentation

Technical Details

• The fix ensures all query functions return consistent dictionary structures when return_dataframe=False
• Added defensive handling in fill_query_results() to gracefully handle both 'rows' and 'data' keys
• No breaking changes to public APIs

Testing

• Verified fix with existing term info serialization tests
• All query functions now return standardized formats for preview result processing

This release improves reliability of term info queries, particularly for individual neurons with connectivity information. Users experiencing preview result errors should see resolution after updating.

VFBquery v0.5.2

Release Notes for VFBquery v0.5.2

Bug Fixes

• Updated term info tests: Fixed assertions in test_term_info_serialization_class and test_term_info_deserialization_from_dict to match updated SOLR data containing additional publications. This prevents CI failures due to evolving external data sources.
• Improved error handling in Owlery queries: Enhanced error differentiation in the Owlery query function to properly handle 400 Bad Request errors separately from other HTTP errors, improving debugging and user experience.

Maintenance

• Updated performance test results across multiple commits to reflect current benchmarks.

Notes

This release focuses on test stability and error handling improvements. The term info test updates ensure compatibility with the latest Virtual Fly Brain data, which has seen additions to publication references.

For full details, see the commit history on GitHub. If you encounter any issues, please report them at https://github.com/VirtualFlyBrain/VFBquery/issues.

VFBquery v0.5.1

VFBquery v0.5.1 Release Notes

Overview

VFBquery v0.5.1 is a maintenance release that builds upon the major performance and feature enhancements introduced in v0.5.0. This release focuses on refining the caching system, improving test coverage, enhancing documentation, fixing minor bugs, and streamlining CI/CD workflows. These improvements ensure better reliability, performance, and developer experience for VirtualFlyBrain data exploration.

🚀 Major Features

Enhanced Caching System

• Background Caching: Implemented background caching for expensive queries in SolrResultCache to improve responsiveness
• Advanced Result Handling: Enhanced cached result handling to support slicing for dict types with 'rows' and 'queries' keys
• Force Refresh Option: Added force refresh capability to caching functions for bypassing cache when needed
• Comprehensive SOLR Caching: Extended SOLR-based caching to similarity queries, term info retrieval, and neuron-related functions
• Unified Caching Architecture: Refactored caching implementation to use SOLR-based caching throughout the codebase for better performance and simplified management

🔧 Improvements

Testing & Quality Assurance

• Expanded Test Coverage: Added initial test cases for VFB term and instance retrieval functions
• Improved Test Logic: Refactored neuron part tests to set minimum expected counts and enhance cache key handling for DataFrame results
• SOLR Availability Handling: Added helper methods to skip tests when SOLR server is unavailable
• Test Result Management: Refactored test results update to use JSON format for better data handling and fixed update logic for appending new templates

Documentation & Examples

• Enhanced README: Added detailed metadata and example queries for JRC2018Unisex brain template, refactored medulla examples, and improved overall clarity
• Consistent Formatting: Refactored README thumbnails for consistent escaping and added sorting functions for data rows in test examples
• Code Quality: Updated test scripts and README for improved functionality and maintainability

Code Quality & Maintenance

• Import Refactoring: Updated import statements to use relative paths and improved code structure readability
• Parser Enhancements: Refactored readme_parser to remove regex for boolean conversion and added update_readme script for JSON block replacement
• Code Block Processing: Added logic to skip processing Python code blocks containing import statements

🐛 Bug Fixes

• Debug Cleanup: Removed debug print statements from cached functions and Owlery client
• URL Corrections: Fixed license icon URL to use HTTP and updated term info serialization test URLs to HTTPS
• Workflow Fixes: Restored original print statements for SOLR unavailability in example execution
• Query Consistency: Refactored neuron region connectivity queries to use 'region' instead of 'label' for clarity in preview columns

🔄 CI/CD Enhancements

• SOLR Integration: Added SOLR availability checks to GitHub Actions workflows with proper exception handling
• Workflow Optimization: Reordered dependency installation steps and removed duplicate conditions in examples workflow
• Test Result Updates: Added automated steps to update performance test results in CI/CD pipeline
• Path Corrections: Fixed paths for updating test results in GitHub Actions workflow

📊 Performance Metrics

• Caching Efficiency: Improved cache hit rates through enhanced SOLR-based caching implementation
• Background Processing: Reduced blocking operations with background caching for expensive queries
• Test Reliability: Better handling of external service dependencies (SOLR) in automated testing

📦 Dependencies

• No dependency changes in this release
• Enhanced compatibility with existing VFB infrastructure through improved error handling

🙏 Acknowledgments

This release includes contributions from the VirtualFlyBrain development team, focusing on user feedback regarding performance, reliability, and usability. Special thanks to automated testing and CI/CD improvements that ensure consistent quality.

---

Installation:

pip install vfbquery==0.5.1

Full Changelog: v0.5.0...v0.5.1

v0.5.0

VFBquery v0.5.0 Release Notes

Overview

VFBquery v0.5.0 introduces significant performance improvements, new connectivity queries, enhanced caching capabilities, and comprehensive test coverage. This release focuses on making VFBquery more robust, faster, and feature-complete for VirtualFlyBrain data exploration.

🚀 Major Features

New Connectivity Queries

• Neuron-Neuron Connectivity: Query neurons connected to a given neuron with synaptic weight information
• Neuron-Region Connectivity: Find brain regions where neurons have synaptic terminals
• Enhanced OWL-based Queries: Improved implementation of lineage clones, fasciculating neurons, and innervating tracts/nerves

Performance Enhancements

• Parallel Processing: Concurrent query execution in fill_query_results for faster term info retrieval
• Extended Timeouts: Increased HTTP request timeouts from 20 to 40 minutes for complex OWL reasoning queries
• SOLR Timeout Increase: Extended SOLR request timeout from 30 to 120 seconds
• Optimized Caching: Smart cache extraction for limited result queries from full cached datasets

Advanced Caching System

• Multi-layer Caching: Combined SOLR and memory caching for optimal performance
• Cache Validation: Enhanced validation to prevent caching of error results
• Preview Parameter Support: Intelligent caching based on preview requirements
• Fallback Mechanisms: Graceful degradation when Neo4j is unavailable

🔧 Improvements

Query Enhancements

• Standardized field naming across all connectivity queries (using label consistently)
• Improved Cypher query patterns for better performance
• Enhanced error handling and retry logic for connection issues
• Better IRI handling and URL construction for OWL queries

Testing & Quality

• Comprehensive test suites for all major query types
• Performance benchmarking with configurable thresholds
• Enhanced CI/CD workflows with proper environment handling
• Mocked testing support for development environments

Documentation

• Updated README with detailed examples and improved formatting
• Comprehensive VFB Queries Reference documentation
• Enhanced API documentation and usage examples
• Performance test reporting and analysis

🐛 Bug Fixes

• Fixed caching issues with error results
• Improved error handling in Owlery client
• Corrected query parameter handling and validation
• Enhanced debug logging for troubleshooting

📊 Performance Metrics

• Significant reduction in query execution times through parallel processing
• Improved cache hit rates with enhanced validation
• Better handling of slow queries with extended timeouts
• Optimized memory usage in caching system

🔄 Migration Notes

• Breaking Change: Connectivity queries now use label field consistently instead of descriptive field names
• Timeout Changes: Default timeouts have increased - update client code if you have custom timeout handling
• Cache Behavior: Enhanced caching may change performance characteristics for repeated queries

📦 Dependencies

• Enhanced compatibility with latest VFB infrastructure
• Improved error handling for network connectivity issues
• Better integration with VFB Connect services

🙏 Acknowledgments

This release includes contributions from the VirtualFlyBrain team and community feedback. Special thanks to users who provided performance feedback and bug reports that helped shape these improvements.

---

Installation:

pip install vfbquery==0.5.0

Full Changelog: v0.4.1...v0.5.0

v0.4.1

What's Changed

• Adding initialize_vfb_connect by @Robbie1977 in #30
• Fix for bracket encoding in label by @Robbie1977 in #31
• fix for encoding in thumbnails by @Robbie1977 in #32
• Fix for testing and encoding by @Robbie1977 in #33

Full Changelog: v0.4.0...v0.4.1

v0.4.0

What's Changed

• Cache Improvements by @Robbie1977 in #29
  This implements comprehensive caching improvements for VFBquery to address cold start performance issues and reduce response times from minutes to seconds. The changes introduce multiple layers of caching, including in-memory, disk-based, and SOLR-based caching systems, all enabled by default with production-ready configurations.

Key changes:

Added SOLR-based result caching with 3-month TTL for server-side optimization
Implemented multi-layer memory and disk caching with intelligent LRU eviction
Enhanced get_instances function with fallback SOLR implementation when Neo4j is unavailable
Added comprehensive performance testing infrastructure and monitoring

Full Changelog: v0.3.4...v0.4.0