Amazon EMR on EKS now supports Spark SQL through the StartJobRun API - devamazonaws.blogspot.com

We are excited to announce that Amazon EMR on EKS release 6.7.0 and onwards includes the ability to run Apache Spark SQL scripts through the StartJobRun API. Spark SQL is a Spark module for structured data processing. Unlike the Spark DataFrame API, Spark SQL interfaces provide Spark with more information about the structure of both the data and computation being performed. Internally, Spark SQL uses this extra information to perform additional optimizations. With this release, you can run Spark SQL queries and Spark SQL-based ETL pipelines directly through Amazon EMR on EKS’ StartJobRun API.

Post Updated on September 19, 2022 at 06:58PM

Comments

Post a Comment

Popular posts from this blog

[MS] Pulling a single item from a C++ parameter pack by its index, remarks - devamazonaws.blogspot.com

In my earlier discussion of pulling a single item from a C++ parameter pack by its index , I noted that with the Pack Indexing proposal, you would be able to write this to pull an item from a parameter pack while preserving its reference category. template<int index, typename...Args> void example(Args&&... args) { auto&& arg = (Args...[index]&&)args...[index]; using Arg = Args...[index]&&; } Why did I need to apply the (Args...[index]&&) cast ? Why can't I just write this: auto&& arg = args...[index]; or possibly decltype(auto) arg = args...[index]; Well, when you write the name of a variable, the result is an lvalue reference, even if the variable is an rvalue reference . Watch: struct S { S(S&); // construct from lvalue S(S&&); // construct from rvalue }; void whathappens(S&& s) { S t = s; // which will it use? } Try it out in your favorite compiler. This code construc...
Read more

[MS] Boosting Azure DevOps Security with GHAS Code Scanning - devamazonaws.blogspot.com

Image
Debjyoti Ganguly shares insights on the security benefits and configuration of GHAS Code Scanning with Azure DevOps. Boosting Azure DevOps Security with GHAS Code Scanning Code scanning, a pipeline-based tool available in GitHub Advanced Security, is designed to detect code vulnerabilities and bugs within the source code of ADO (Azure DevOps) repositories. Utilizing CodeQL as a static analysis tool, it performs query analysis and variant analysis. When vulnerabilities are found, it generates security alerts. CodeQL CodeQL is a powerful static analysis tool used for showing vulnerabilities and bugs in source code. It enables developers to write custom queries that analyze codebases, searching for specific patterns and potential security issues. By converting code into a database format, CodeQL allows for sophisticated, database-like queries to detect flaws. CodeQL in Action 1. Preparing the Code Create a CodeQL Database : Extract and structure the code into a database for analy...
Read more

[MS] Going beyond the empty set: Embracing the power of other empty things - devamazonaws.blogspot.com

The empty set contains nothing. This sounds really silly, but it's actually really nice. The Windows Runtime has a policy that if a method returns a collection (such as an IVector ), and the method produces no results, then it should return an empty collection, rather than a null reference . That way, consumers can just iterate over the collection without having to deal with a null test. For example, suppose you have a method Widget:: Get­Associated­Doodads which returns an IVectorView<Doodad> representing the Doodad objects that have been associated with a Widget object. If no Doodad s have been associated with the Widget , then it should return an empty vector, not a null pointer. That allows developers to write the natural-looking code: // C# foreach (var doodad in widget.GetAssociatedDoodads()) { ⟦ process each doodad ⟧ } // C++/WinRT for (auto&& doodad : widget.GetAssociatedDoodads()) { ⟦ process each doodad ⟧ } // JavaScript widget.GetAssociated...
Read more