-
Notifications
You must be signed in to change notification settings - Fork 136
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RUMM-334 Benchmark merged into Integration #63
Conversation
Unlike we discussed before, we don't actually need In fact, workspace would make things more complex as embedded projects work as |
Good idea 👌. This will also speed up CI by running tests in only one project. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The general idea looks great 👌. However, I think we should not give too much details in CONTRIBUTING.md
as we'll have to always keep it up to date. Rather, we may want to dispatch those details to separate README.md
files in particular folders.
6fe0273
to
a699311
Compare
Whatever works is good for me 👌. The only thing we must guarantee is that |
a699311
to
58504b4
Compare
58504b4
to
b95e1bd
Compare
404a72a
to
26e9020
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need to also:
- update paths in
sources.swiftlint.yml
- update paths in
tests.swiftlint.yml
- delete those references from
Integration.xcodeproj
, as they are invalid:
- configure all tests to use "Release" configuration - "Debug" doesn't make sense for Benchmarks:
Likewise:
- let's provision physical devices for running the tests by conditionally importing
dependency-manager-tests/xcconfig/app-target.xcconfig
intests-specific-config.xcconfig
and resetting all the "Signing" Build Settings to default, so Xcode will use the values fromxcconfig
. - how about rearranging following groups so they are more aligned with the setup? I think we can have three main groups:
Benchmark
,Integration
,Helpers
.
``` | ||
|
||
This will install `swiftlint` and configure custom Datadog file templates for Xcode. Also, `examples-secret.xcconfig` file will be created - update it with a client token obtained on Datadog website. | ||
### Repo structure |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's keep it up-to-date with a current layout. So maybe we can remove those changes now and add them iteratively with every PR about the layout change, WDYT?
@@ -7,6 +7,11 @@ | |||
import Foundation | |||
@testable import Datadog | |||
|
|||
func clearPersistedLogs() throws { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This function doesn't follow the way we handle Directory
and duplicates the code of .delete()
. We could basically expose new directory on global constant and leverage already existing delete()
implementation:
let logsDirectory = try! Directory(withSubdirectoryPath: LogsPersistenceStrategy.Constants.logFilesSubdirectory)
Then, basically logsDirectory.delete()
can be called wherever we need.
This is similar pattern to what we do for:
let temporaryDirectory = obtainUniqueTemporaryDirectory()
@@ -104,33 +103,11 @@ workflows: | |||
inputs: | |||
- scheme: Integration | |||
- simulator_device: iPhone 11 | |||
- is_clean_build: 'yes' | |||
- is_clean_build: 'no' | |||
- should_build_before_test: 'no' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why no clean_build
? Does it make any difference on CI?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that test
target doesn't have anything to build
, when we call build
or even clean
it fails with an error along the lines of "this scheme is not configured for running"
with these parameters we run test
without explicit build/clean
, which works ✅
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
); | ||
inputPaths = ( | ||
); | ||
name = "⚙️ Run linter"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We also need ⚙️ Run linter
Build Phase to be re-done in Integration.xcodeproj
.
@@ -23,6 +19,8 @@ class BenchmarkTests: XCTestCase { | |||
super.setUp() | |||
if server == nil { server = try! setUpMockServerConnection() } | |||
if serverSession == nil { serverSession = server.obtainUniqueRecordingSession() } | |||
|
|||
temporaryDirectory.delete() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, I don't think we need this. Both integration tests and benchmarks do not write to temporaryDirectory
🤔, no?
import HTTPServerMock | ||
import XCTest | ||
|
||
class LoggingIntegrationTests: IntegrationTests { | ||
override func setUp() { | ||
super.setUp() | ||
try! clearPersistedLogs() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we can do it in base IntegrationTests
? This way we won't need @testable import Datadog
in this file, and we'll make sure we're accessing only public APIs. WDYT?
|
26e9020
to
d0eb26a
Compare
As suggested in #79 maybe it's worth to consider following for this PR:
func testWhenRunningAppTarget_allLogsAreUploadedToServer() {
let app = XCUIApplication()
app.launch()
Thread.sleep(...)
// set of assertions on data delivered to mock server
} This will very powerful enough to fulfill upcoming needs:
WDYT @buranmert ? |
52c475c
to
2214d42
Compare
Integration and Benchmark projects are removed
2214d42
to
5cb2251
Compare
|
👍 OK, it sounds fine to bring this within example project.
But we also need provisioning to run the integration tests locally 🙌 . Without this, there's no use of benchmark tests and we won't be able to compare the SDK performance in any way. Also, this part of integration test can't be run anywhere: #if targetEnvironment(simulator)
// When running on iOS Simulator
matcher.assertNoValue(forKey: LogMatcher.JSONKey.mobileNetworkCarrierName)
matcher.assertNoValue(forKey: LogMatcher.JSONKey.mobileNetworkCarrierISOCountryCode)
matcher.assertNoValue(forKey: LogMatcher.JSONKey.mobileNetworkCarrierRadioTechnology)
matcher.assertNoValue(forKey: LogMatcher.JSONKey.mobileNetworkCarrierAllowsVoIP)
#else
// When running on physical device with SIM card registered
matcher.assertValue(forKeyPath: LogMatcher.JSONKey.mobileNetworkCarrierName, isTypeOf: String.self)
matcher.assertValue(forKeyPath: LogMatcher.JSONKey.mobileNetworkCarrierISOCountryCode, isTypeOf: String.self)
matcher.assertValue(forKeyPath: LogMatcher.JSONKey.mobileNetworkCarrierRadioTechnology, isTypeOf: String.self)
matcher.assertValue(forKeyPath: LogMatcher.JSONKey.mobileNetworkCarrierAllowsVoIP, isTypeOf: Bool.self)
#endif Before this PR, I used to provision integration and benchmark tests manually. Also the benchmark baseline for my iPhone X was checked into the repo. This file should be re-created for each device we own, so we still have baselines to compare the SDK performance. Now in this PR, it's easy to provide this provisioning, as set of proper |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Few last small comments, the rest looks 💯
CONTRIBUTING.md
Outdated
|
||
Unit tests are part of the `Datadog.xcodeproj` project. Integration tests and benchmarks are located in separate projects in `instrumented-tests` folder. | ||
Isolated example apps using `cocoapods`, `carthage` and `spm` to ensure SDK is well integrated in all platforms |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isolated example apps using `cocoapods`, `carthage` and `spm` to ensure SDK is well integrated in all platforms | |
Isolated example apps using `cocoapods`, `carthage` and `spm` to ensure SDK is well integrated with all supported dependency managers. |
@@ -0,0 +1 @@ | |||
MOCK_SERVER_ADDRESS= |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file should be git-ignored, otherwise will be producing git changes each time when integration tests are run.
Makefile
Outdated
@@ -31,7 +31,7 @@ examples: | |||
@echo "OK 👌" | |||
|
|||
benchmark: | |||
@cd instrumented-tests/Benchmark && $(MAKE) | |||
@xcodebuild -workspace "Datadog.xcworkspace" -scheme "DatadogIntegrationTests" test -destination "name=iPhone 11 Pro Max" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we still need make benchmark
? This tests are run through bitrise.yml
in much nicier way with xcpretty
log. Also we duplicate the iPhone 11 Pro Max
instrumentation here, I think it's better to centralize it with bitrise.yml
only. Now there's IMO no need to do anything for integration tests in Makefile
.
|
||
import Foundation | ||
|
||
func clearPersistedLogs() throws { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see any use of this in this PR - shouldn't this be removed 🤔?
} | ||
|
||
override func tearDown() { | ||
try! FileManager.default.removeItem(at: directory.url) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Isn't this just
try! FileManager.default.removeItem(at: directory.url) | |
self.directory.delete() |
now?
5cb2251
to
c25278a
Compare
ca0ba8e
to
86e1690
Compare
86e1690
to
76e74c2
Compare
What and why?
We'd like to simplify the repo layout and make it easier to find things for new developers.
How?
Integration
andBenchmark
projects from the repoDatadogIntegrationTests
inDatadog.xcodeproj
Review checklist