AWS Neuron adds support for PyTorch 2.1 and Llama-2-70b model inference - devamazonaws.blogspot.com

AWS Neuron is the SDK for Amazon EC2 Inferentia and Trainium based instances purpose-built for generative AI. Today, with Neuron 2.16 release, we are announcing support for Llama-2 70b model inference on Inf2 instances. 

Post Updated on December 22, 2023 at 06:22PM

Comments

Post a Comment

Popular posts from this blog

[MS] Pulling a single item from a C++ parameter pack by its index, remarks - devamazonaws.blogspot.com

In my earlier discussion of pulling a single item from a C++ parameter pack by its index , I noted that with the Pack Indexing proposal, you would be able to write this to pull an item from a parameter pack while preserving its reference category. template<int index, typename...Args> void example(Args&&... args) { auto&& arg = (Args...[index]&&)args...[index]; using Arg = Args...[index]&&; } Why did I need to apply the (Args...[index]&&) cast ? Why can't I just write this: auto&& arg = args...[index]; or possibly decltype(auto) arg = args...[index]; Well, when you write the name of a variable, the result is an lvalue reference, even if the variable is an rvalue reference . Watch: struct S { S(S&); // construct from lvalue S(S&&); // construct from rvalue }; void whathappens(S&& s) { S t = s; // which will it use? } Try it out in your favorite compiler. This code construc...
Read more

[MS] Debugger breakpoints are usually implemented by patching the in-memory copy of the code - devamazonaws.blogspot.com

In practice, when you set a code breakpoint in the debugger, the debugger replaces the instruction at that location with a breakpoint instruction.¹ When execution reaches that instruction, it will encounter the breakpoint instruction and break into the debugger. When the program has been stopped in the debugger, what happens next can vary from debugger to debugger. Some debuggers remove all their breakpoints when the program stops, and then restore the breakpoints when the program resumes. Other debuggers leave the breakpoints in place even when the program is stopped. In both cases, if you inspect the memory in the debugger, you will see the original unpatched code. In the first case, it's because the code really is unpatched; the breakpoint instructions are removed. In the second case, it's because the debugger is lying to you and showing you the original bytes even though they aren't what are in memory right now. Most of the time, this deception is insignificant. Eve...
Read more

[MS] The case of the crash when destructing a std::map - devamazonaws.blogspot.com

A customer reported that they were getting crashes while destructing a std::map . Here's the point of the crash: eax=245bd25c ebx=00004d33 ecx=31feece4 edx=00c40000 esi=00000000 edi=31feece4 eip=563397db esp=245bd250 ebp=245bd268 iopl=0 nv up ei ng nz na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010286 contoso!std::_Tree<⟦...⟧>::~_Tree+0x2b: 563397db cmp byte ptr [esi+0Dh],0 ds:002b:0000000d=?? 0:054> k9 ChildEBP RetAddr (Inline) -------- contoso!std::_Tree_val<⟦...⟧>::_Erase_tree [xtree @ 1073] (Inline) -------- contoso!std::_Tree_val<⟦...⟧>::_Erase_head+0x5 [xtree @ 1073] 245bd268 56338313 contoso!std::_Tree<⟦...⟧>::~_Tree+0x2b [xtree @ 1073] 245bd290 56362695 contoso!Contoso::IdCollection::~IdCollection+0x163 245bd2b4 564c8510 contoso!Contoso::LogEntry::~LogEntry+0xd5 245bd2e0 5691a999 contoso!Contoso::LogEntries::Cleanup+0x90 (Inline) -------- con...
Read more