Elastic 7.11 released: General availability of searchable snapshots and the new cold tier, and the beta of schema on read

We are pleased to announce the general availability (GA) of Elastic 7.11. This release brings a broad set of new capabilities to our Elastic Enterprise Search, Observability, and Security solutions, which are built into the Elastic Stack — Elasticsearch and Kibana. This release enables customers to optimize for cost, performance, insight, and flexibility with the general availability of searchable snapshots and the beta of schema on read. The beta of a new web crawler in Elastic Enterprise Search makes content from publicly accessible web sites easily searchable. Elastic Observability delivers enhanced root cause analysis, troubleshooting, and application observability with service health and host detail views. New prebuilt detection rules and machine learning jobs and customizable alert notifications in Elastic Security improve detection and remediation, and a unified analyst workspace streamlines SecOps.  We are also announcing key improvements to Elastic Cloud, the best managed Elastic service and the only one that includes our solutions. Elastic Cloud offers rich support for searchable snapshots, autoscaling of data and machine learning nodes, and increased availability and better search performance with enhanced cross-cluster replication (CCR) and cross-cluster search (CCS). Elastic 7.11 is available now on Elastic Cloud — the only hosted Elasticsearch offering to include all of the new features in this latest release. You can also download the Elastic Stack and our cloud orchestration products, Elastic Cloud Enterprise and Elastic Cloud for Kubernetes, for a self-managed experience.

As previously announced, Elastic is changing the licensing options for Elasticsearch and Kibana with the 7.11 release. We are moving the Apache 2 licensed code to be dual licensed under both the Elastic License and SSPL. We have also made significant updates to the Elastic License to simplify it and make it more permissive. Our distribution and the source code for all of our free and paid features are available under the Elastic License v2, with the source code of a core set of our free features also available under SSPL v1. This change does not impact any of our customers or the vast majority of our community. Read on for the key release highlights. To get the full feature rundown, dive into the individual solution and product blog posts. Elastic StackRetain and search more data with searchable snapshots on low-cost object stores and the new cold data tier.Searchable snapshots let you search across data in snapshots stored on low-cost object storage such as AWS S3, Microsoft Azure Storage, and Google Cloud Storage, transforming how you can balance storage costs, search performance, and depth of insight from the data in your Elasticsearch cluster. Searchable snapshots can significantly reduce storage costs; they support a new cold tier capability, which is now generally available and also available in Elastic Cloud, cutting infrastructure costs by up to 50% with minimal performance impact. Leverage schema on read with runtime fields, giving you the choice of flexibility and cost efficiency with schema on read or blazing fast performance with schema on write, all in one stack.Runtime fields give you the ability to define the schema for your index at query time. This new capability, in beta in 7.11, allows you to discover new data and new workflows by creating a schema on the fly, giving you unprecedented flexibility to find new insights while choosing how to trade off cost and performance.  Elasticsearch is known for being a blazing fast distributed search and analytics engine because data is stored in structured indices created when data is written to disk, or schema on write. This organized structure requires understanding and planning for how data will be represented in Elasticsearch, but the big payoff is speed, scale, and relevance. But there are times when you need a way to explore and examine data in a new way without planning the data schema in advance. Create fluid data structures at the time of search with schema on read using runtime fields. This flexibility reduces the time to first insight, but with some trade-off on overall performance. The Elastic approach gives you the agility to choose between schema on write and schema on read based on your search scenario. Runtime fields are initially supported in Elasticsearch with this release, and we plan to extend support throughout Kibana. Schema on Read is in beta in 7.11. Read more about it in the dedicated runtime fields blog. Create, manage, and monitor alerts and notifications across the Elastic Stack and external systems with a new generally available alerting framework.Knowing when something important is happening within your digital ecosystem is mission-critical regardless of the kind of work you do. From threat detections to application performance warnings and to physical asset tracking, receiving timely alerts when a major change occurs is critical to acting on data insights. Eight months ago we launched the beta of a new alerting framework to do exactly this inside of the Elastic Stack, and, with the release of 7.11, this new alerting framework is generally available.  We have seen tremendous community enthusiasm and adoption during the beta, which has reinforced our commitment to building a framework that is deeply integrated into every solution within the Elastic Stack, easy to manage centrally, and focused not only on supporting detection, but also driving action and integrating Elastic directly into your workflows. The alerting interface is incorporated directly into the Elastic Security and Elastic Observability solutions, and we have expanded the framework to include third-party alert integrations with platforms like PagerDuty, ServiceNow, and Microsoft Teams. Alerts can be easily governed using role-based access controls.

The alerting framework helps drive workflow and collaborationRead about these features and more in the Kibana 7.11 blog and the Elasticsearch 7.11 blog. Elastic Enterprise SearchMake content from publicly accessible web sites easily searchable with the new web crawler for Elastic App Search.Making content searchable takes many forms. Elastic App Search already lets users ingest content via uploading or pasting JSON and through API endpoints. With Elastic Enterprise Search 7.11, users can now ingest content with a powerful web crawler that retrieves information from publicly accessible web sites, making content easily searchable in your App Search engines. As with any ingestion method on App Search, the schema is inferred upon ingestion and can be updated in near real time with one click. With clicks — not code — users can customize the web crawler rules so entry points can be specified while exclusion rules instruct the web crawler on pages, content, and terms to avoid.

The new web crawler in Elastic App Search Search content in Box, a leading cloud content management system, with Elastic Workplace Search.As one of the early pioneers in cloud-based storage, Box has evolved into a leading cloud content management system with millions of global users. Elastic Enterprise Search now supports Box as a content source inside Workplace Search. The prebuilt connector includes document-level permissions so the right user sees what they’re supposed to see — and nothing else. The addition of Box expands an already robust portfolio of content sources available in Workplace Search, including Google Drive and Dropbox. Extend granular access controls with document-level permissions for Atlassian Jira Cloud and Confluence Cloud for Elastic Workplace Search.Not all content is created — or shared — equally. Sensitive and private content needs to be shared with explicitly defined individuals or groups, with document-level access to these files becoming even more critical when they’re easily available to be searched. Elastic Workplace Search now includes document-level permissions for Atlassian’s Jira Cloud and Confluence Cloud so the same permissions set in these source applications are inherited by Elastic Workplace Search. Get a scoop on all of the new Elastic Enterprise Search features in the Elastic Enterprise Search 7.11 release blog. Elastic ObservabilityAccelerate root cause analysis and troubleshooting with the new service health view in Elastic APM. Modern cloud-native applications are typically made up of hundreds of microservices, and the ability to quickly pinpoint the performance and health of an individual service is critical to incident investigation workflows. The new service overview page summarizes all the information about the health of a service in one place and makes it easier for developers and SREs to troubleshoot performance issues. Time series charts of service latency, traffic, and error rate provide a high-level view of service KPIs over time. Overlaid annotations such as deployment markers and anomaly alerts provide rich context into key events that might have contributed to changes in behavior. The service overview page uses sparklines to provide a compact view into temporal trends of subcomponents, making it easy to spot unusual changes in behavior and drive investigations. The service overview page also shows service health broken down by underlying infrastructure instances (e.g., containers) that the service is deployed on, so that you can connect issues to problems with underlying infrastructure. Future releases will bring additional context and views into the mix to further streamline and accelerate troubleshooting and root cause analysis workflows.

The new service health view in Elastic APM Troubleshoot infrastructure issues faster with a new host details view in Elastic Metrics.The resource heatmap in the Elastic Metrics app helps you spot trouble in your infrastructure and narrow down next steps of an investigation. A new view in the metrics UI makes it easy to go from the high-level view to inspect what is happening on an individual host. Clicking on a tile in the heatmap brings up a popup window that surfaces key information, including time charts of key host metrics, logs generated by the host, processes running