replica
/
hanchenye-llvm-project
2
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
hanchenye-llvm-project/llvm/lib/XRay/RecordInitializer.cpp

418 lines
17 KiB
C++
Raw Normal View History

[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
//===- FDRRecordProducer.cpp - XRay FDR Mode Record Producer --------------===//
//
Update the file headers across all of the LLVM projects in the monorepo to reflect the new license. We understand that people may be surprised that we're moving the header entirely to discuss the new license. We checked this carefully with the Foundation's lawyer and we believe this is the correct approach. Essentially, all code in the project is now made available by the LLVM project under our new license, so you will see that the license headers include that license only. Some of our contributors have contributed code under our old license, and accordingly, we have retained a copy of our old license notice in the top-level files in each project and repository. llvm-svn: 351636
2019-01-19 16:50:56 +08:00
// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
// See https://llvm.org/LICENSE.txt for license information.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
//
//===----------------------------------------------------------------------===//
#include "llvm/XRay/FDRRecords.h"
namespace llvm {
namespace xray {
Error RecordInitializer::visit(BufferExtents &R) {
if (!E.isValidOffsetForDataOfSize(OffsetPtr, sizeof(uint64_t)))
return createStringError(std::make_error_code(std::errc::bad_address),
"Invalid offset for a buffer extent (%d).",
OffsetPtr);
auto PreReadOffset = OffsetPtr;
R.Size = E.getU64(&OffsetPtr);
if (PreReadOffset == OffsetPtr)
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
return createStringError(std::make_error_code(std::errc::invalid_argument),
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
"Cannot read buffer extent at offset %d.",
OffsetPtr);
OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - PreReadOffset);
return Error::success();
}
Error RecordInitializer::visit(WallclockRecord &R) {
if (!E.isValidOffsetForDataOfSize(OffsetPtr,
MetadataRecord::kMetadataBodySize))
return createStringError(std::make_error_code(std::errc::bad_address),
"Invalid offset for a wallclock record (%d).",
OffsetPtr);
auto BeginOffset = OffsetPtr;
auto PreReadOffset = OffsetPtr;
R.Seconds = E.getU64(&OffsetPtr);
if (OffsetPtr == PreReadOffset)
return createStringError(
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
std::make_error_code(std::errc::invalid_argument),
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
"Cannot read wall clock 'seconds' field at offset %d.", OffsetPtr);
PreReadOffset = OffsetPtr;
R.Nanos = E.getU32(&OffsetPtr);
if (OffsetPtr == PreReadOffset)
return createStringError(
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
std::make_error_code(std::errc::invalid_argument),
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
"Cannot read wall clock 'nanos' field at offset %d.", OffsetPtr);
// Align to metadata record size boundary.
assert(OffsetPtr - BeginOffset <= MetadataRecord::kMetadataBodySize);
OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - BeginOffset);
return Error::success();
}
Error RecordInitializer::visit(NewCPUIDRecord &R) {
if (!E.isValidOffsetForDataOfSize(OffsetPtr,
MetadataRecord::kMetadataBodySize))
return createStringError(std::make_error_code(std::errc::bad_address),
"Invalid offset for a new cpu id record (%d).",
OffsetPtr);
[XRay] Add TSC to NewCPUId Records Summary: This more correctly reflects the data written by the FDR mode runtime. This is a continuation of the work in D50441. Reviewers: mboerger, eizan Subscribers: hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51911 llvm-svn: 341905
2018-09-11 14:36:51 +08:00
auto BeginOffset = OffsetPtr;
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
auto PreReadOffset = OffsetPtr;
R.CPUId = E.getU16(&OffsetPtr);
if (OffsetPtr == PreReadOffset)
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
return createStringError(std::make_error_code(std::errc::invalid_argument),
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
"Cannot read CPU id at offset %d.", OffsetPtr);
[XRay] Add TSC to NewCPUId Records Summary: This more correctly reflects the data written by the FDR mode runtime. This is a continuation of the work in D50441. Reviewers: mboerger, eizan Subscribers: hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51911 llvm-svn: 341905
2018-09-11 14:36:51 +08:00
PreReadOffset = OffsetPtr;
R.TSC = E.getU64(&OffsetPtr);
if (OffsetPtr == PreReadOffset)
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
return createStringError(std::make_error_code(std::errc::invalid_argument),
[XRay] Add TSC to NewCPUId Records Summary: This more correctly reflects the data written by the FDR mode runtime. This is a continuation of the work in D50441. Reviewers: mboerger, eizan Subscribers: hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51911 llvm-svn: 341905
2018-09-11 14:36:51 +08:00
"Cannot read CPU TSC at offset %d.", OffsetPtr);
OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - BeginOffset);
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
return Error::success();
}
Error RecordInitializer::visit(TSCWrapRecord &R) {
if (!E.isValidOffsetForDataOfSize(OffsetPtr,
MetadataRecord::kMetadataBodySize))
return createStringError(std::make_error_code(std::errc::bad_address),
"Invalid offset for a new TSC wrap record (%d).",
OffsetPtr);
auto PreReadOffset = OffsetPtr;
R.BaseTSC = E.getU64(&OffsetPtr);
if (PreReadOffset == OffsetPtr)
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
return createStringError(std::make_error_code(std::errc::invalid_argument),
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
"Cannot read TSC wrap record at offset %d.",
OffsetPtr);
OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - PreReadOffset);
return Error::success();
}
Error RecordInitializer::visit(CustomEventRecord &R) {
if (!E.isValidOffsetForDataOfSize(OffsetPtr,
MetadataRecord::kMetadataBodySize))
return createStringError(std::make_error_code(std::errc::bad_address),
"Invalid offset for a custom event record (%d).",
OffsetPtr);
auto BeginOffset = OffsetPtr;
auto PreReadOffset = OffsetPtr;
R.Size = E.getSigned(&OffsetPtr, sizeof(int32_t));
if (PreReadOffset == OffsetPtr)
return createStringError(
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
std::make_error_code(std::errc::invalid_argument),
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
"Cannot read a custom event record size field offset %d.", OffsetPtr);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
if (R.Size <= 0)
return createStringError(
std::make_error_code(std::errc::bad_address),
"Invalid size for custom event (size = %d) at offset %d.", R.Size,
OffsetPtr);
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
PreReadOffset = OffsetPtr;
R.TSC = E.getU64(&OffsetPtr);
if (PreReadOffset == OffsetPtr)
return createStringError(
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
std::make_error_code(std::errc::invalid_argument),
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
"Cannot read a custom event TSC field at offset %d.", OffsetPtr);
[XRay] Add CPU ID in Custom Event FDR Records Summary: This change cuts across compiler-rt and llvm, to increment the FDR log version number to 4, and include the CPU ID in the custom event records. This is a step towards allowing us to change the `llvm::xray::Trace` object to start representing both custom and typed events in the stream of records. Follow-on changes will allow us to change the kinds of records we're presenting in the stream of traces, to incorporate the data in custom/typed events. A follow-on change will handle the typed event case, where it may not fit within the 15-byte buffer for metadata records. This work is part of the larger effort to enable writing analysis and processing tools using a common in-memory representation of the events found in traces. The work will focus on porting existing tools in LLVM to use the common representation and informing the design of a library/framework for expressing trace event analysis as C++ programs. Reviewers: mboerger, eizan Subscribers: hiraditya, mgrang, llvm-commits Differential Revision: https://reviews.llvm.org/D53920 llvm-svn: 345798
2018-11-01 08:18:52 +08:00
// For version 4 onwards, of the FDR log, we want to also capture the CPU ID
// of the custom event.
if (Version >= 4) {
PreReadOffset = OffsetPtr;
R.CPU = E.getU16(&OffsetPtr);
if (PreReadOffset == OffsetPtr)
return createStringError(
std::make_error_code(std::errc::invalid_argument),
"Missing CPU field at offset %d", OffsetPtr);
}
assert(OffsetPtr > BeginOffset &&
OffsetPtr - BeginOffset <= MetadataRecord::kMetadataBodySize);
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - BeginOffset);
// Next we read in a fixed chunk of data from the given offset.
if (!E.isValidOffsetForDataOfSize(OffsetPtr, R.Size))
return createStringError(
std::make_error_code(std::errc::bad_address),
"Cannot read %d bytes of custom event data from offset %d.", R.Size,
OffsetPtr);
std::vector<uint8_t> Buffer;
Buffer.resize(R.Size);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
PreReadOffset = OffsetPtr;
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
if (E.getU8(&OffsetPtr, Buffer.data(), R.Size) != Buffer.data())
[XRay] Use TSC delta encoding for custom/typed events Summary: This change updates the version number for FDR logs to 5, and update the trace processing to support changes in the custom event records. In the runtime, since we're already writing down the record preamble to handle CPU migrations and TSC wraparound, we can use the same TSC delta encoding in the custom event and typed event records that we use in function event records. We do the same change to typed events (which were unsupported before this change in the trace processing) which now show up in the trace. Future changes should increase our testing coverage to make custom and typed events as first class entities in the FDR mode log processing tools. This change is also a good example of how we end up supporting new record types in the FDR mode implementation. This shows the places where new record types are added and supported. Depends on D54139. Reviewers: mboerger Subscribers: hiraditya, arphaman, jfb, llvm-commits Differential Revision: https://reviews.llvm.org/D54140 llvm-svn: 346293
2018-11-07 12:37:42 +08:00
return createStringError(
std::make_error_code(std::errc::invalid_argument),
"Failed reading data into buffer of size %d at offset %d.", R.Size,
OffsetPtr);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
assert(OffsetPtr >= PreReadOffset);
if (OffsetPtr - PreReadOffset != static_cast<uint32_t>(R.Size))
return createStringError(
std::make_error_code(std::errc::invalid_argument),
"Failed reading enough bytes for the custom event payload -- read %d "
"expecting %d bytes at offset %d.",
OffsetPtr - PreReadOffset, R.Size, PreReadOffset);
[XRay] Use TSC delta encoding for custom/typed events Summary: This change updates the version number for FDR logs to 5, and update the trace processing to support changes in the custom event records. In the runtime, since we're already writing down the record preamble to handle CPU migrations and TSC wraparound, we can use the same TSC delta encoding in the custom event and typed event records that we use in function event records. We do the same change to typed events (which were unsupported before this change in the trace processing) which now show up in the trace. Future changes should increase our testing coverage to make custom and typed events as first class entities in the FDR mode log processing tools. This change is also a good example of how we end up supporting new record types in the FDR mode implementation. This shows the places where new record types are added and supported. Depends on D54139. Reviewers: mboerger Subscribers: hiraditya, arphaman, jfb, llvm-commits Differential Revision: https://reviews.llvm.org/D54140 llvm-svn: 346293
2018-11-07 12:37:42 +08:00
R.Data.assign(Buffer.begin(), Buffer.end());
return Error::success();
}
Error RecordInitializer::visit(CustomEventRecordV5 &R) {
if (!E.isValidOffsetForDataOfSize(OffsetPtr,
MetadataRecord::kMetadataBodySize))
return createStringError(std::make_error_code(std::errc::bad_address),
"Invalid offset for a custom event record (%d).",
OffsetPtr);
auto BeginOffset = OffsetPtr;
auto PreReadOffset = OffsetPtr;
R.Size = E.getSigned(&OffsetPtr, sizeof(int32_t));
if (PreReadOffset == OffsetPtr)
return createStringError(
std::make_error_code(std::errc::invalid_argument),
"Cannot read a custom event record size field offset %d.", OffsetPtr);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
if (R.Size <= 0)
return createStringError(
std::make_error_code(std::errc::bad_address),
"Invalid size for custom event (size = %d) at offset %d.", R.Size,
OffsetPtr);
[XRay] Use TSC delta encoding for custom/typed events Summary: This change updates the version number for FDR logs to 5, and update the trace processing to support changes in the custom event records. In the runtime, since we're already writing down the record preamble to handle CPU migrations and TSC wraparound, we can use the same TSC delta encoding in the custom event and typed event records that we use in function event records. We do the same change to typed events (which were unsupported before this change in the trace processing) which now show up in the trace. Future changes should increase our testing coverage to make custom and typed events as first class entities in the FDR mode log processing tools. This change is also a good example of how we end up supporting new record types in the FDR mode implementation. This shows the places where new record types are added and supported. Depends on D54139. Reviewers: mboerger Subscribers: hiraditya, arphaman, jfb, llvm-commits Differential Revision: https://reviews.llvm.org/D54140 llvm-svn: 346293
2018-11-07 12:37:42 +08:00
PreReadOffset = OffsetPtr;
R.Delta = E.getSigned(&OffsetPtr, sizeof(int32_t));
if (PreReadOffset == OffsetPtr)
return createStringError(
std::make_error_code(std::errc::invalid_argument),
"Cannot read a custom event record TSC delta field at offset %d.",
OffsetPtr);
assert(OffsetPtr > BeginOffset &&
OffsetPtr - BeginOffset <= MetadataRecord::kMetadataBodySize);
OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - BeginOffset);
// Next we read in a fixed chunk of data from the given offset.
if (!E.isValidOffsetForDataOfSize(OffsetPtr, R.Size))
return createStringError(
std::make_error_code(std::errc::bad_address),
"Cannot read %d bytes of custom event data from offset %d.", R.Size,
OffsetPtr);
std::vector<uint8_t> Buffer;
Buffer.resize(R.Size);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
PreReadOffset = OffsetPtr;
[XRay] Use TSC delta encoding for custom/typed events Summary: This change updates the version number for FDR logs to 5, and update the trace processing to support changes in the custom event records. In the runtime, since we're already writing down the record preamble to handle CPU migrations and TSC wraparound, we can use the same TSC delta encoding in the custom event and typed event records that we use in function event records. We do the same change to typed events (which were unsupported before this change in the trace processing) which now show up in the trace. Future changes should increase our testing coverage to make custom and typed events as first class entities in the FDR mode log processing tools. This change is also a good example of how we end up supporting new record types in the FDR mode implementation. This shows the places where new record types are added and supported. Depends on D54139. Reviewers: mboerger Subscribers: hiraditya, arphaman, jfb, llvm-commits Differential Revision: https://reviews.llvm.org/D54140 llvm-svn: 346293
2018-11-07 12:37:42 +08:00
if (E.getU8(&OffsetPtr, Buffer.data(), R.Size) != Buffer.data())
return createStringError(
std::make_error_code(std::errc::invalid_argument),
"Failed reading data into buffer of size %d at offset %d.", R.Size,
OffsetPtr);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
assert(OffsetPtr >= PreReadOffset);
if (OffsetPtr - PreReadOffset != static_cast<uint32_t>(R.Size))
return createStringError(
std::make_error_code(std::errc::invalid_argument),
"Failed reading enough bytes for the custom event payload -- read %d "
"expecting %d bytes at offset %d.",
OffsetPtr - PreReadOffset, R.Size, PreReadOffset);
[XRay] Use TSC delta encoding for custom/typed events Summary: This change updates the version number for FDR logs to 5, and update the trace processing to support changes in the custom event records. In the runtime, since we're already writing down the record preamble to handle CPU migrations and TSC wraparound, we can use the same TSC delta encoding in the custom event and typed event records that we use in function event records. We do the same change to typed events (which were unsupported before this change in the trace processing) which now show up in the trace. Future changes should increase our testing coverage to make custom and typed events as first class entities in the FDR mode log processing tools. This change is also a good example of how we end up supporting new record types in the FDR mode implementation. This shows the places where new record types are added and supported. Depends on D54139. Reviewers: mboerger Subscribers: hiraditya, arphaman, jfb, llvm-commits Differential Revision: https://reviews.llvm.org/D54140 llvm-svn: 346293
2018-11-07 12:37:42 +08:00
R.Data.assign(Buffer.begin(), Buffer.end());
return Error::success();
}
Error RecordInitializer::visit(TypedEventRecord &R) {
if (!E.isValidOffsetForDataOfSize(OffsetPtr,
MetadataRecord::kMetadataBodySize))
return createStringError(std::make_error_code(std::errc::bad_address),
"Invalid offset for a typed event record (%d).",
OffsetPtr);
auto BeginOffset = OffsetPtr;
auto PreReadOffset = OffsetPtr;
R.Size = E.getSigned(&OffsetPtr, sizeof(int32_t));
if (PreReadOffset == OffsetPtr)
return createStringError(
std::make_error_code(std::errc::invalid_argument),
"Cannot read a typed event record size field offset %d.", OffsetPtr);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
if (R.Size <= 0)
return createStringError(
std::make_error_code(std::errc::bad_address),
"Invalid size for typed event (size = %d) at offset %d.", R.Size,
OffsetPtr);
[XRay] Use TSC delta encoding for custom/typed events Summary: This change updates the version number for FDR logs to 5, and update the trace processing to support changes in the custom event records. In the runtime, since we're already writing down the record preamble to handle CPU migrations and TSC wraparound, we can use the same TSC delta encoding in the custom event and typed event records that we use in function event records. We do the same change to typed events (which were unsupported before this change in the trace processing) which now show up in the trace. Future changes should increase our testing coverage to make custom and typed events as first class entities in the FDR mode log processing tools. This change is also a good example of how we end up supporting new record types in the FDR mode implementation. This shows the places where new record types are added and supported. Depends on D54139. Reviewers: mboerger Subscribers: hiraditya, arphaman, jfb, llvm-commits Differential Revision: https://reviews.llvm.org/D54140 llvm-svn: 346293
2018-11-07 12:37:42 +08:00
PreReadOffset = OffsetPtr;
R.Delta = E.getSigned(&OffsetPtr, sizeof(int32_t));
if (PreReadOffset == OffsetPtr)
return createStringError(
std::make_error_code(std::errc::invalid_argument),
"Cannot read a typed event record TSC delta field at offset %d.",
OffsetPtr);
PreReadOffset = OffsetPtr;
R.EventType = E.getU16(&OffsetPtr);
if (PreReadOffset == OffsetPtr)
return createStringError(
std::make_error_code(std::errc::invalid_argument),
"Cannot read a typed event record type field at offset %d.", OffsetPtr);
assert(OffsetPtr > BeginOffset &&
OffsetPtr - BeginOffset <= MetadataRecord::kMetadataBodySize);
OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - BeginOffset);
// Next we read in a fixed chunk of data from the given offset.
if (!E.isValidOffsetForDataOfSize(OffsetPtr, R.Size))
return createStringError(
std::make_error_code(std::errc::bad_address),
"Cannot read %d bytes of custom event data from offset %d.", R.Size,
OffsetPtr);
std::vector<uint8_t> Buffer;
Buffer.resize(R.Size);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
PreReadOffset = OffsetPtr;
[XRay] Use TSC delta encoding for custom/typed events Summary: This change updates the version number for FDR logs to 5, and update the trace processing to support changes in the custom event records. In the runtime, since we're already writing down the record preamble to handle CPU migrations and TSC wraparound, we can use the same TSC delta encoding in the custom event and typed event records that we use in function event records. We do the same change to typed events (which were unsupported before this change in the trace processing) which now show up in the trace. Future changes should increase our testing coverage to make custom and typed events as first class entities in the FDR mode log processing tools. This change is also a good example of how we end up supporting new record types in the FDR mode implementation. This shows the places where new record types are added and supported. Depends on D54139. Reviewers: mboerger Subscribers: hiraditya, arphaman, jfb, llvm-commits Differential Revision: https://reviews.llvm.org/D54140 llvm-svn: 346293
2018-11-07 12:37:42 +08:00
if (E.getU8(&OffsetPtr, Buffer.data(), R.Size) != Buffer.data())
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
return createStringError(
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
std::make_error_code(std::errc::invalid_argument),
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
"Failed reading data into buffer of size %d at offset %d.", R.Size,
OffsetPtr);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
assert(OffsetPtr >= PreReadOffset);
if (OffsetPtr - PreReadOffset != static_cast<uint32_t>(R.Size))
return createStringError(
std::make_error_code(std::errc::invalid_argument),
"Failed reading enough bytes for the typed event payload -- read %d "
"expecting %d bytes at offset %d.",
OffsetPtr - PreReadOffset, R.Size, PreReadOffset);
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
R.Data.assign(Buffer.begin(), Buffer.end());
return Error::success();
}
Error RecordInitializer::visit(CallArgRecord &R) {
if (!E.isValidOffsetForDataOfSize(OffsetPtr,
MetadataRecord::kMetadataBodySize))
return createStringError(std::make_error_code(std::errc::bad_address),
"Invalid offset for a call argument record (%d).",
OffsetPtr);
auto PreReadOffset = OffsetPtr;
R.Arg = E.getU64(&OffsetPtr);
if (PreReadOffset == OffsetPtr)
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
return createStringError(std::make_error_code(std::errc::invalid_argument),
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
"Cannot read a call arg record at offset %d.",
OffsetPtr);
OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - PreReadOffset);
return Error::success();
}
Error RecordInitializer::visit(PIDRecord &R) {
if (!E.isValidOffsetForDataOfSize(OffsetPtr,
MetadataRecord::kMetadataBodySize))
return createStringError(std::make_error_code(std::errc::bad_address),
"Invalid offset for a process ID record (%d).",
OffsetPtr);
auto PreReadOffset = OffsetPtr;
[XRay] Update RecordInitializer for PIDRecord Since we changed the storage for the PID in PIDRecord instances, we need to also update the way we load the data from a DataExtractor through the RecordInitializer. llvm-svn: 341243
2018-09-01 04:02:55 +08:00
R.PID = E.getSigned(&OffsetPtr, 4);
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
if (PreReadOffset == OffsetPtr)
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
return createStringError(std::make_error_code(std::errc::invalid_argument),
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
"Cannot read a process ID record at offset %d.",
OffsetPtr);
OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - PreReadOffset);
return Error::success();
}
Error RecordInitializer::visit(NewBufferRecord &R) {
if (!E.isValidOffsetForDataOfSize(OffsetPtr,
MetadataRecord::kMetadataBodySize))
return createStringError(std::make_error_code(std::errc::bad_address),
"Invalid offset for a new buffer record (%d).",
OffsetPtr);
auto PreReadOffset = OffsetPtr;
R.TID = E.getSigned(&OffsetPtr, sizeof(int32_t));
if (PreReadOffset == OffsetPtr)
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
return createStringError(std::make_error_code(std::errc::invalid_argument),
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
"Cannot read a new buffer record at offset %d.",
OffsetPtr);
OffsetPtr += MetadataRecord::kMetadataBodySize - (OffsetPtr - PreReadOffset);
return Error::success();
}
Error RecordInitializer::visit(EndBufferRecord &R) {
if (!E.isValidOffsetForDataOfSize(OffsetPtr,
MetadataRecord::kMetadataBodySize))
return createStringError(std::make_error_code(std::errc::bad_address),
"Invalid offset for an end-of-buffer record (%d).",
OffsetPtr);
OffsetPtr += MetadataRecord::kMetadataBodySize;
return Error::success();
}
Error RecordInitializer::visit(FunctionRecord &R) {
// For function records, we need to retreat one byte back to read a full
// unsigned 32-bit value. The first four bytes will have the following
// layout:
//
// bit 0 : function record indicator (must be 0)
// bits 1..3 : function record type
// bits 4..32 : function id
//
if (OffsetPtr == 0 || !E.isValidOffsetForDataOfSize(
--OffsetPtr, FunctionRecord::kFunctionRecordSize))
return createStringError(std::make_error_code(std::errc::bad_address),
"Invalid offset for a function record (%d).",
OffsetPtr);
auto BeginOffset = OffsetPtr;
auto PreReadOffset = BeginOffset;
uint32_t Buffer = E.getU32(&OffsetPtr);
if (PreReadOffset == OffsetPtr)
return createStringError(std::make_error_code(std::errc::bad_address),
"Cannot read function id field from offset %d.",
OffsetPtr);
[XRay] Improve FDR trace handling and error messaging Summary: This change covers a number of things spanning LLVM and compiler-rt, which are related in a non-trivial way. In LLVM, we have a library that handles the FDR mode even log loading, which uses C++'s runtime polymorphism feature to better faithfully represent the events that are written down by the FDR mode runtime. We do this by interpreting a trace that's serliased in a common format agreed upon by both the trace loading library and the FDR mode runtime. This library is under active development, which consists of features allowing us to reconstitute a higher-level event log. This event log is used by the conversion and visualisation tools we have for interpreting XRay traces. One of the tools we have is a diagnostic tool in llvm-xray called `fdr-dump` which we've been using to debug our expectations of what the FDR runtime should be writing and what the logical FDR event log structures are. We use this fairly extensively to reason about why some non-trivial traces we're generating with FDR mode runtimes fail to convert or fail to parse correctly. One of these failures we've found in manual debugging of some of the traces we've seen involve an inconsistency between the buffer extents (a record indicating how many bytes to follow are part of a logical thread's event log) and the record of the bytes written into the log -- sometimes it turns out the data could be garbage, due to buffers being recycled, but sometimes we're seeing the buffer extent indicating a log is "shorter" than the actual records associated with the buffer. This case happens particularly with function entry records with a call argument. This change for now updates the FDR mode runtime to write the bytes for the function call and arg record before updating the buffer extents atomically, allowing multiple threads to see a consistent view of the data in the buffer using the atomic counter associated with a buffer. What we're trying to prevent here is partial updates where we see the intermediary updates to the buffer extents (function record size then call argument record size) becoming observable from another thread, for instance, one doing the serialization/flushing. To do both diagnose this issue properly, we need to be able to honour the extents being set in the `BufferExtents` records marking the beginning of the logical buffers when reading an FDR trace. Since LLVM doesn't use C++'s RTTI mechanism, we instead follow the advice in the documentation for LLVM Style RTTI (https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html). We then rely on this RTTI feature to ensure that our file-based record producer (our streaming "deserializer") can honour the extents of individual buffers as we interpret traces. This also sets us up to be able to eventually do smart skipping/continuation of FDR logs, seeking instead to find BufferExtents records in cases where we find potentially recoverable errors. In the meantime, we make this change to operate in a strict mode when reading logical buffers with extent records. Reviewers: mboerger Subscribers: hiraditya, llvm-commits, jfb Differential Revision: https://reviews.llvm.org/D54201 llvm-svn: 346473
2018-11-09 14:26:48 +08:00
// To get the function record type, we shift the buffer one to the right
// (truncating the function record indicator) then take the three bits
// (0b0111) to get the record type as an unsigned value.
unsigned FunctionType = (Buffer >> 1) & 0x07u;
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
switch (FunctionType) {
case static_cast<unsigned>(RecordTypes::ENTER):
case static_cast<unsigned>(RecordTypes::ENTER_ARG):
case static_cast<unsigned>(RecordTypes::EXIT):
case static_cast<unsigned>(RecordTypes::TAIL_EXIT):
R.Kind = static_cast<RecordTypes>(FunctionType);
break;
default:
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
return createStringError(std::make_error_code(std::errc::invalid_argument),
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
"Unknown function record type '%d' at offset %d.",
FunctionType, BeginOffset);
}
R.FuncId = Buffer >> 4;
PreReadOffset = OffsetPtr;
R.Delta = E.getU32(&OffsetPtr);
if (OffsetPtr == PreReadOffset)
[XRay] Use std::errc::invalid_argument instead of std::errc::bad_message This change should appease the mingw32 builds. Similar to r293725. Differential Revision: https://reviews.llvm.org/D53742 llvm-svn: 345416
2018-10-27 02:25:27 +08:00
return createStringError(std::make_error_code(std::errc::invalid_argument),
[XRay] FDRTraceWriter and FDR Trace Loading Summary: This is the first step in the larger refactoring and reduction of D50441. This step in the process does the following: - Introduces more granular types of `Record`s representing the many kinds of records written/read by the Flight Data Recorder (FDR) mode `Trace` loading function(s). - Introduces an abstract `RecordVisitor` type meant to handle the processing of the various `Record` derived types. This `RecordVisitor` has two implementations in this patch: `RecordInitializer` and `FDRTraceWriter`. - We also introduce a convenience interface for building a collection of `Record` instances called a `LogBuilder`. This allows us to generate sequences of `Record` instances manually (used in unit tests but useful otherwise). - The`FDRTraceWriter` class implements the `RecordVisitor` interface and handles the writing of metadata records to a `raw_ostream`. We demonstrate that in the unit test, we can generate in-memory FDR mode traces using the specific `Record` derived types, which we load through the `loadTrace(...)` function yielding valid `Trace` objects. This patch introduces the required types and concepts for us to start replacing the logic implemented in the `loadFDRLog` function to use the more granular types. In subsequent patches, we will introduce more visitor implementations which isolate the verification, printing, indexing, production/consumption, and finally the conversion of the FDR mode logs. The overarching goal of these changes is to make handling FDR mode logs better tested, more understandable, more extensible, and more systematic. This will also allow us to better represent the execution trace, as we improve the fidelity of the events we represent in an XRay `Trace` object, which we intend to do after FDR mode log processing is in better shape. Reviewers: eizan Reviewed By: eizan Subscribers: mgorny, hiraditya, llvm-commits Differential Revision: https://reviews.llvm.org/D51210 llvm-svn: 341029
2018-08-30 15:22:21 +08:00
"Failed reading TSC delta from offset %d.",
OffsetPtr);
assert(FunctionRecord::kFunctionRecordSize == (OffsetPtr - BeginOffset));
return Error::success();
}
} // namespace xray
} // namespace llvm